2024-04-13T03:08:46.767 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-13T03:08:46.767 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242 --name vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} --owner scheduled_vshankar@teuthology -- /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/orig.config.yaml 2024-04-13T03:08:46.826 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-13T03:08:46.923 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242 branch: wip-vshankar-testing-20240412.042930 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} email: vshankar@redhat.com first_in_suite: false job_id: '7654242' kernel: client: branch: testing flavor: default kdb: true sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240412.042930 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240412.042930 sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 owner: scheduled_vshankar@teuthology priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7517 sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 sleep_before_teardown: 0 subset: 55/1024 suite: fs suite_branch: wip-vshankar-testing-20240412.042930 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_218a691417ef57c5d22a8d5cf3c5f41f7b542838/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN2HJad3UK2cBZqWdWwY2LbNC8GcXG99vL24lyk30Is/SsXNDVuMnYZHJSRl8whGXG6DoJRrO9NSa4KD3lS1woQ= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5f3IpZEfv1nVd7oXygMcKV7gLar+slPYHKPjyrp613EACIIN9rajZtVcHCdyTAgk2/cFKn+gjWIVsK8A3KPdY= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEP5S2oScurDuzg9YvE0nHj9+UqNo3pg9v+QZo/5JKTcZozh1+YnVjpf7V2j1oQicQ8rJESmTWTrLEMuVNrjBJc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-12_17:34:28 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-13T03:08:46.923 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_218a691417ef57c5d22a8d5cf3c5f41f7b542838/qa; will attempt to use it 2024-04-13T03:08:46.924 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_218a691417ef57c5d22a8d5cf3c5f41f7b542838/qa/tasks 2024-04-13T03:08:46.924 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-13T03:08:46.926 INFO:teuthology.task.internal:Checking packages... 2024-04-13T03:08:46.950 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '218a691417ef57c5d22a8d5cf3c5f41f7b542838' 2024-04-13T03:08:46.951 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-13T03:08:46.951 INFO:teuthology.packaging:ref: None 2024-04-13T03:08:46.951 INFO:teuthology.packaging:tag: None 2024-04-13T03:08:46.951 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240412.042930 2024-04-13T03:08:46.951 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:08:46.951 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240412.042930 2024-04-13T03:08:47.107 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2884.g218a6914 2024-04-13T03:08:47.109 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-13T03:08:47.115 INFO:teuthology.task.internal:no buildpackages task found 2024-04-13T03:08:47.115 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-13T03:08:47.132 INFO:teuthology.task.internal:Saving configuration 2024-04-13T03:08:47.153 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-13T03:08:47.159 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T03:08:47.185 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 03:03:38.063441', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN2HJad3UK2cBZqWdWwY2LbNC8GcXG99vL24lyk30Is/SsXNDVuMnYZHJSRl8whGXG6DoJRrO9NSa4KD3lS1woQ='} 2024-04-13T03:08:47.204 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 03:03:38.062041', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5f3IpZEfv1nVd7oXygMcKV7gLar+slPYHKPjyrp613EACIIN9rajZtVcHCdyTAgk2/cFKn+gjWIVsK8A3KPdY='} 2024-04-13T03:08:47.227 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 03:03:38.064737', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEP5S2oScurDuzg9YvE0nHj9+UqNo3pg9v+QZo/5JKTcZozh1+YnVjpf7V2j1oQicQ8rJESmTWTrLEMuVNrjBJc='} 2024-04-13T03:08:47.227 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-13T03:08:47.233 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-13T03:08:47.233 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-13T03:08:47.233 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-13T03:08:47.233 INFO:teuthology.run_tasks:Running task console_log... 2024-04-13T03:08:47.338 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe90fdf9160>, signals=[15]) 2024-04-13T03:08:47.339 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-13T03:08:47.345 INFO:teuthology.task.internal:Opening connections... 2024-04-13T03:08:47.345 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T03:08:47.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T03:08:47.455 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-04-13T03:08:47.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T03:08:47.541 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-04-13T03:08:47.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T03:08:47.631 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-13T03:08:47.637 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-04-13T03:08:47.656 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-04-13T03:08:47.656 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-04-13T03:08:47.714 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-13T03:08:47.715 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-04-13T03:08:47.715 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-13T03:08:47.715 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-13T03:08:47.715 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-13T03:08:47.715 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-04-13T03:08:47.740 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-04-13T03:08:47.758 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-04-13T03:08:47.758 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-04-13T03:08:47.814 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-13T03:08:47.815 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-13T03:08:47.816 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-04-13T03:08:47.832 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-04-13T03:08:47.850 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-04-13T03:08:47.850 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-04-13T03:08:47.906 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="9" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="9" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el9" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:LOGO="fedora-logo-icon" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-13T03:08:47.907 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-13T03:08:47.908 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-04-13T03:08:47.932 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-13T03:08:47.943 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-13T03:08:47.956 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-13T03:08:47.956 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-04-13T03:08:47.959 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-04-13T03:08:47.965 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-04-13T03:08:47.981 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-13T03:08:47.987 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-13T03:08:47.987 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-04-13T03:08:48.016 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-04-13T03:08:48.018 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-04-13T03:08:48.111 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-13T03:08:48.189 INFO:teuthology.run_tasks:Running task kernel... 2024-04-13T03:08:48.204 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'} 2024-04-13T03:08:48.205 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-13T03:08:48.205 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-13T03:08:48.205 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-13T03:08:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T03:08:48.222 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-13T03:08:48.473 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-13T03:08:48.473 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-13T03:08:48.473 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-04-13T03:08:48.493 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-361.el9.x86_64 2024-04-13T03:08:48.493 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-13T03:08:48.494 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-13T03:08:48.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum upgrade -y linux-firmware 2024-04-13T03:08:50.637 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - AppStream 17 MB/s | 8.1 MB 00:00 2024-04-13T03:08:52.305 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - BaseOS 8.4 MB/s | 2.5 MB 00:00 2024-04-13T03:08:53.489 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-13T03:08:55.579 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-13T03:09:01.989 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.2 MB 00:02 2024-04-13T03:09:04.573 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-04-13T03:09:06.443 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-13T03:09:11.666 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-04-13T03:09:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:Modular dependency problems: 2024-04-13T03:09:14.041 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-13T03:09:14.041 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 1: conflicting requests 2024-04-13T03:09:14.041 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 2: conflicting requests 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 3: conflicting requests 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 4: conflicting requests 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 5: conflicting requests 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 6: conflicting requests 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 7: conflicting requests 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.042 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 8: conflicting requests 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 9: conflicting requests 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 10: conflicting requests 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 11: conflicting requests 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 12: conflicting requests 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 13: conflicting requests 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 14: conflicting requests 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.043 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 15: conflicting requests 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 16: conflicting requests 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 17: conflicting requests 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 18: conflicting requests 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 19: conflicting requests 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 20: conflicting requests 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 21: conflicting requests 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.044 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 22: conflicting requests 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 23: conflicting requests 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 24: conflicting requests 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 25: conflicting requests 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 26: conflicting requests 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 27: conflicting requests 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 28: conflicting requests 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 29: conflicting requests 2024-04-13T03:09:14.045 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 30: conflicting requests 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 31: conflicting requests 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 32: conflicting requests 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 33: conflicting requests 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 34: conflicting requests 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 35: conflicting requests 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 36: conflicting requests 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.046 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 37: conflicting requests 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 39: conflicting requests 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.047 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.048 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.049 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.050 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T03:09:14.552 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: linux-firmware noarch 20240219-143.el9 baseos 376 M 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 1 Package 2024-04-13T03:09:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:09:14.564 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 376 M 2024-04-13T03:09:14.564 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-13T03:09:23.733 INFO:teuthology.orchestra.run.smithi016.stdout:linux-firmware-20240219-143.el9.noarch.rpm 42 MB/s | 376 MB 00:08 2024-04-13T03:09:23.733 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-13T03:09:23.733 INFO:teuthology.orchestra.run.smithi016.stdout:Total 41 MB/s | 376 MB 00:09 2024-04-13T03:09:26.921 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-13T03:09:27.025 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-13T03:09:27.025 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-13T03:09:30.401 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-13T03:09:30.402 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-13T03:09:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-13T03:09:38.097 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-13T03:09:38.206 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-13T03:09:39.402 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-13T03:09:39.403 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-13T03:09:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-13T03:09:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:09:39.793 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-04-13T03:09:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-13T03:09:39.793 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:09:39.793 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-13T03:09:40.097 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-13T03:09:40.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-13T03:09:40.245 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-13T03:09:40.246 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-13T03:09:40.248 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi016.front.sepia.ceph.com to complete... 2024-04-13T03:09:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13 03:09:52 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-13T03:09:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:FINISHED --2024-04-13 03:09:52-- 2024-04-13T03:09:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:Total wall clock time: 13s 2024-04-13T03:09:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:Downloaded: 1 files, 341M in 12s (27.6 MB/s) 2024-04-13T03:09:52.990 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-13T03:09:52.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-13T03:09:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying... ######################################## 2024-04-13T03:09:55.198 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing... ######################################## 2024-04-13T03:09:55.283 INFO:teuthology.orchestra.run.smithi016.stdout:Updating / installing... 2024-04-13T03:10:03.678 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-13T03:10:20.635 INFO:teuthology.orchestra.run.smithi016.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-13T03:10:44.501 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-13T03:10:44.502 DEBUG:teuthology.orchestra.run.smithi016:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-13T03:10:44.588 INFO:teuthology.orchestra.run.smithi016.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:10:44.588 INFO:teuthology.orchestra.run.smithi016.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-13T03:10:44.589 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-13T03:10:44.590 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-13T03:10:44.591 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-13T03:10:44.592 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-13T03:10:44.593 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-13T03:10:44.594 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-13T03:10:44.595 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-13T03:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-13T03:10:44.597 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-13T03:10:44.598 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-13T03:10:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-13T03:10:44.600 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-13T03:10:44.601 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-13T03:10:44.602 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-13T03:10:44.603 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-13T03:10:44.604 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-13T03:10:44.605 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-13T03:10:44.606 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-13T03:10:44.607 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-13T03:10:44.608 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-13T03:10:44.609 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-13T03:10:44.610 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-13T03:10:44.611 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-13T03:10:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-13T03:10:44.613 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-13T03:10:44.614 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-13T03:10:44.615 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-13T03:10:44.616 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-13T03:10:44.617 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-13T03:10:44.618 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-13T03:10:44.619 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-13T03:10:44.620 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-13T03:10:44.621 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-13T03:10:44.622 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-13T03:10:44.623 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-13T03:10:44.624 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-13T03:10:44.625 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-13T03:10:44.626 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-13T03:10:44.627 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-13T03:10:44.628 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-13T03:10:44.629 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-13T03:10:44.630 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-13T03:10:44.631 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-13T03:10:44.632 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-13T03:10:44.633 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-13T03:10:44.634 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-13T03:10:44.635 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-13T03:10:44.636 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-13T03:10:44.637 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-13T03:10:44.638 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-13T03:10:44.639 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-13T03:10:44.640 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-13T03:10:44.641 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-13T03:10:44.642 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-13T03:10:44.643 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-13T03:10:44.644 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-13T03:10:44.645 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-13T03:10:44.646 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-13T03:10:44.647 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-13T03:10:44.648 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-13T03:10:44.649 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-13T03:10:44.650 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-13T03:10:44.651 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-13T03:10:44.652 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-13T03:10:44.653 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-13T03:10:44.654 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-13T03:10:44.655 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-13T03:10:44.656 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-13T03:10:44.657 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-13T03:10:44.658 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-13T03:10:44.659 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-13T03:10:44.660 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-13T03:10:44.661 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-13T03:10:44.662 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-13T03:10:44.663 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-13T03:10:44.664 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-13T03:10:44.665 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-13T03:10:44.666 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-13T03:10:44.667 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-13T03:10:44.668 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-13T03:10:44.669 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-13T03:10:44.670 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-13T03:10:44.671 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-13T03:10:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-13T03:10:44.673 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-13T03:10:44.674 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-13T03:10:44.675 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-13T03:10:44.676 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-13T03:10:44.677 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-13T03:10:44.678 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-13T03:10:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-13T03:10:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-13T03:10:44.681 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-13T03:10:44.682 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-13T03:10:44.683 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-13T03:10:44.684 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-13T03:10:44.685 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-13T03:10:44.686 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-13T03:10:44.687 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-13T03:10:44.688 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-13T03:10:44.689 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-13T03:10:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-13T03:10:44.691 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-13T03:10:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-13T03:10:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-13T03:10:44.694 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-13T03:10:44.695 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-13T03:10:44.696 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-13T03:10:44.697 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-13T03:10:44.698 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-13T03:10:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-13T03:10:44.700 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-13T03:10:44.701 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-13T03:10:44.702 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-13T03:10:44.703 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-13T03:10:44.704 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-13T03:10:44.705 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-13T03:10:44.706 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-13T03:10:44.707 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-13T03:10:44.708 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-13T03:10:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-13T03:10:44.710 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-13T03:10:44.711 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-13T03:10:44.712 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-13T03:10:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-13T03:10:44.714 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-13T03:10:44.715 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-13T03:10:44.716 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-13T03:10:44.717 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-13T03:10:44.718 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-13T03:10:44.719 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-13T03:10:44.720 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-13T03:10:44.721 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-13T03:10:44.722 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-13T03:10:44.723 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-13T03:10:44.724 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-13T03:10:44.725 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-13T03:10:44.726 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-13T03:10:44.727 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-13T03:10:44.728 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-13T03:10:44.729 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-13T03:10:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-13T03:10:44.731 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-13T03:10:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-13T03:10:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-13T03:10:44.734 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-13T03:10:44.735 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-13T03:10:44.736 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-13T03:10:44.737 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-13T03:10:44.738 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-13T03:10:44.739 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-13T03:10:44.740 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-13T03:10:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-13T03:10:44.742 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-13T03:10:44.743 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-13T03:10:44.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-13T03:10:44.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-13T03:10:44.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-13T03:10:44.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-13T03:10:44.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-13T03:10:44.744 INFO:teuthology.orchestra.run.smithi016.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-13T03:10:44.748 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:10:44.748 DEBUG:teuthology.orchestra.run.smithi016:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-13T03:10:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:postinstall scriptlet (using /bin/sh): 2024-04-13T03:10:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-13T03:10:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-13T03:10:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:fi 2024-04-13T03:10:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:for file in vmlinuz System.map config; do 2024-04-13T03:10:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-13T03:10:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-13T03:10:44.772 INFO:teuthology.orchestra.run.smithi016.stdout: fi 2024-04-13T03:10:44.772 INFO:teuthology.orchestra.run.smithi016.stdout:done 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:fi 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:fi 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:10:44.773 INFO:teuthology.orchestra.run.smithi016.stdout:fi 2024-04-13T03:10:44.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-04-13T03:10:44.865 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 68.el9 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8264254 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-13T03:10:44.866 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:10:44.867 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-04-13T03:10:44.868 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-13T03:10:44.868 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:10:44.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-13T03:10:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-04-13T03:10:47.002 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-13T03:10:47.038 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-04-13T03:10:47.040 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-04-13T03:10:47.055 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.b5Iho6yO1b 2024-04-13T03:10:47.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.b5Iho6yO1b 2024-04-13T03:10:47.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.b5Iho6yO1b 2024-04-13T03:10:47.284 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.b5Iho6yO1b is 6KB 2024-04-13T03:10:47.333 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.b5Iho6yO1b 2024-04-13T03:10:47.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-04-13T03:10:47.417 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-13T03:10:47.417 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-13T03:10:47.417 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-13T03:10:47.417 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-13T03:10:47.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:10:47.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-04-13T03:11:17.572 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-13T03:11:17.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-13T03:11:17.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T03:12:17.576 DEBUG:teuthology.orchestra.remote:timed out 2024-04-13T03:12:26.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-13T03:12:26.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T03:12:28.003 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-13T03:12:28.798 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-13T03:12:28.798 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-13T03:12:28.799 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-04-13T03:12:28.838 INFO:teuthology.orchestra.run.smithi016.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:12:28.839 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-13T03:12:28.839 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-13T03:12:28.839 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-13T03:12:29.842 DEBUG:teuthology.parallel:result is None 2024-04-13T03:12:29.842 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-13T03:12:29.849 INFO:teuthology.task.internal:Creating test directory... 2024-04-13T03:12:29.850 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-13T03:12:29.858 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-13T03:12:29.860 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-13T03:12:29.896 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-13T03:12:29.903 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-13T03:12:29.913 INFO:teuthology.task.internal:Creating archive directory... 2024-04-13T03:12:29.913 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-13T03:12:29.943 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-13T03:12:29.945 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-13T03:12:29.994 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-13T03:12:30.001 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-13T03:12:30.001 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-13T03:12:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T03:12:30.033 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-13T03:12:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T03:12:30.050 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-13T03:12:30.057 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-13T03:12:30.075 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-13T03:12:30.123 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T03:12:30.128 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T03:12:30.148 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T03:12:30.151 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T03:12:30.210 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T03:12:30.300 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-13T03:12:30.306 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-13T03:12:30.313 INFO:teuthology.task.internal:Configuring sudo... 2024-04-13T03:12:30.313 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-13T03:12:30.319 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-13T03:12:30.322 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-13T03:12:30.392 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-13T03:12:30.401 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-13T03:12:30.401 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-13T03:12:30.405 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-13T03:12:30.408 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-13T03:12:30.440 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T03:12:30.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T03:12:30.658 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T03:12:30.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T03:12:30.818 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:12:30.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-13T03:12:31.020 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T03:12:31.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T03:12:31.127 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T03:12:31.170 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T03:12:31.214 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:12:31.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-13T03:12:31.282 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T03:12:31.327 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-13T03:12:31.383 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T03:12:31.467 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-13T03:12:31.511 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:12:31.511 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-13T03:12:31.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-04-13T03:12:31.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-04-13T03:12:31.587 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-04-13T03:12:31.623 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T03:12:31.657 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T03:12:31.685 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T03:12:32.152 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-13T03:12:32.160 INFO:teuthology.task.internal:Starting timer... 2024-04-13T03:12:32.160 INFO:teuthology.run_tasks:Running task pcp... 2024-04-13T03:12:32.179 INFO:teuthology.run_tasks:Running task selinux... 2024-04-13T03:12:32.188 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-13T03:12:32.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-04-13T03:12:32.313 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-04-13T03:12:32.319 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-04-13T03:12:32.365 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-04-13T03:12:32.367 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-04-13T03:12:32.419 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-04-13T03:12:32.421 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-13T03:12:32.421 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-04-13T03:12:32.473 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-04-13T03:12:32.473 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-04-13T03:12:32.495 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-04-13T03:12:32.495 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-04-13T03:12:32.517 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-04-13T03:12:32.517 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-04-13T03:12:32.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-13T03:12:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T03:12:32.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-13T03:12:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T03:12:32.622 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-13T03:12:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T03:12:32.650 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-13T03:12:32.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-04-13T03:12:32.735 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-04-13T03:12:32.764 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-04-13T03:12:32.793 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-13T03:12:32.803 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-13T03:12:32.809 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-13T03:12:32.956 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-13T03:12:32.970 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-13T03:12:32.971 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi119.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-13T03:18:53.223 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-04-13T03:18:53.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-13T03:18:53.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T03:18:53.377 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-13T03:18:53.546 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-13T03:18:53.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-13T03:18:53.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T03:18:53.638 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-04-13T03:18:53.707 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-13T03:18:53.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-13T03:18:53.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-13T03:18:53.800 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-13T03:18:53.865 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-13T03:18:53.865 INFO:teuthology.run_tasks:Running task clock... 2024-04-13T03:18:53.877 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-13T03:18:53.877 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-13T03:18:53.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T03:18:53.885 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-13T03:18:53.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T03:18:53.888 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-13T03:18:53.888 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T03:18:53.921 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-13T03:18:53.934 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-13T03:18:53.939 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-13T03:18:53.951 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-13T03:18:53.970 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-04-13T03:18:53.979 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-04-13T03:18:53.983 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-04-13T03:18:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-13T03:18:53.993 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-04-13T03:18:54.002 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-13T03:18:54.010 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-13T03:18:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-13T03:18:54.026 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-13T03:18:54.049 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-13T03:18:54.061 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-04-13T03:18:54.064 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T03:18:54.064 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-04-13T03:18:54.064 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.064 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.065 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.065 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.069 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-04-13T03:18:54.074 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T03:18:54.074 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-13T03:18:54.074 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.074 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.074 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.074 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-04-13T03:18:54.183 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-04-13T03:18:54.239 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-13T03:18:54.296 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-13T03:18:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-13T03:18:54.436 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T03:18:54.436 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-13T03:18:54.436 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.436 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.436 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.436 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T03:18:54.438 INFO:teuthology.run_tasks:Running task install... 2024-04-13T03:18:54.449 DEBUG:teuthology.task.install:project ceph 2024-04-13T03:18:54.449 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'}} 2024-04-13T03:18:54.449 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'} 2024-04-13T03:18:54.449 INFO:teuthology.task.install:Using flavor: default 2024-04-13T03:18:54.457 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-13T03:18:54.457 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-13T03:18:54.458 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'tag': None, 'wait_for_package': False} 2024-04-13T03:18:54.458 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:18:54.460 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'tag': None, 'wait_for_package': False} 2024-04-13T03:18:54.460 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:18:54.461 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'tag': None, 'wait_for_package': False} 2024-04-13T03:18:54.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:18:54.603 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/ 2024-04-13T03:18:54.604 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2884.g218a6914 2024-04-13T03:18:54.611 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/ 2024-04-13T03:18:54.611 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2884.g218a6914 2024-04-13T03:18:54.673 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/ 2024-04-13T03:18:54.673 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2884.g218a6914 2024-04-13T03:18:54.725 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-13T03:18:54.725 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:18:54.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-13T03:18:54.782 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-13T03:18:54.783 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:18:54.783 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-13T03:18:54.813 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-13T03:18:54.814 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/218a691417ef57c5d22a8d5cf3c5f41f7b542838/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-13T03:18:54.826 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-13T03:18:54.827 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/218a691417ef57c5d22a8d5cf3c5f41f7b542838/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-13T03:18:54.841 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-13T03:18:54.841 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:18:54.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-13T03:18:54.872 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-13T03:18:54.872 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/218a691417ef57c5d22a8d5cf3c5f41f7b542838/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-13T03:18:54.884 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-13T03:18:54.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-13T03:18:54.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-13T03:18:54.967 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-13T03:18:55.022 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-13T03:18:55.034 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-04-13T03:18:55.035 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-04-13T03:18:55.090 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-04-13T03:18:55.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-04-13T03:18:55.105 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-13T03:18:55.202 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-04-13T03:18:55.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-04-13T03:18:55.273 INFO:teuthology.orchestra.run.smithi188.stdout:74 files removed 2024-04-13T03:18:55.300 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-13T03:18:55.329 INFO:teuthology.orchestra.run.smithi119.stdout:74 files removed 2024-04-13T03:18:55.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-13T03:18:55.580 INFO:teuthology.orchestra.run.smithi016.stdout:74 files removed 2024-04-13T03:18:55.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-13T03:18:56.074 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 194 kB/s | 83 kB 00:00 2024-04-13T03:18:56.111 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 160 kB/s | 83 kB 00:00 2024-04-13T03:18:56.499 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-13T03:18:56.520 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-13T03:18:56.532 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 216 kB/s | 83 kB 00:00 2024-04-13T03:18:56.800 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 7.0 kB/s | 1.9 kB 00:00 2024-04-13T03:18:56.888 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-13T03:18:57.141 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 22 kB/s | 12 kB 00:00 2024-04-13T03:18:57.335 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 16 MB/s | 8.1 MB 00:00 2024-04-13T03:18:57.365 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - AppStream 18 MB/s | 8.1 MB 00:00 2024-04-13T03:18:57.666 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-04-13T03:18:58.319 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-04-13T03:18:58.953 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 8.5 MB/s | 2.5 MB 00:00 2024-04-13T03:18:59.024 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.5 MB 00:00 2024-04-13T03:18:59.910 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-13T03:19:00.081 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-13T03:19:00.254 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.5 MB 00:00 2024-04-13T03:19:01.299 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-13T03:19:01.840 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-04-13T03:19:05.849 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-13T03:19:07.275 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 16 kB 00:00 2024-04-13T03:19:09.059 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-13T03:19:14.217 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-04-13T03:19:16.779 INFO:teuthology.orchestra.run.smithi188.stderr:Modular dependency problems: 2024-04-13T03:19:16.779 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-13T03:19:16.779 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 1: conflicting requests 2024-04-13T03:19:16.779 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 2: conflicting requests 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 3: conflicting requests 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 4: conflicting requests 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 5: conflicting requests 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 6: conflicting requests 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 7: conflicting requests 2024-04-13T03:19:16.780 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 8: conflicting requests 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 9: conflicting requests 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 10: conflicting requests 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 11: conflicting requests 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 12: conflicting requests 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.781 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 13: conflicting requests 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 14: conflicting requests 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 15: conflicting requests 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 16: conflicting requests 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 17: conflicting requests 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 18: conflicting requests 2024-04-13T03:19:16.782 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 19: conflicting requests 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 20: conflicting requests 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 21: conflicting requests 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 22: conflicting requests 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 23: conflicting requests 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.783 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 24: conflicting requests 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 25: conflicting requests 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 26: conflicting requests 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 27: conflicting requests 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 28: conflicting requests 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 29: conflicting requests 2024-04-13T03:19:16.784 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.785 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 30: conflicting requests 2024-04-13T03:19:16.785 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.785 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 31: conflicting requests 2024-04-13T03:19:16.785 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.785 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 32: conflicting requests 2024-04-13T03:19:16.785 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.785 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 33: conflicting requests 2024-04-13T03:19:16.785 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.785 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 34: conflicting requests 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 35: conflicting requests 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 36: conflicting requests 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 37: conflicting requests 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.786 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 39: conflicting requests 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.787 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T03:19:16.788 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.788 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.788 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.788 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.788 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T03:19:16.788 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.789 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.789 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.789 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.789 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T03:19:16.789 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.789 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.789 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.789 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.789 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T03:19:16.790 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.791 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.791 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T03:19:16.791 INFO:teuthology.orchestra.run.smithi188.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T03:19:17.197 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-13T03:19:17.198 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-13T03:19:17.212 INFO:teuthology.orchestra.run.smithi188.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-13T03:19:17.213 INFO:teuthology.orchestra.run.smithi188.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-13T03:19:17.214 INFO:teuthology.orchestra.run.smithi188.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-13T03:19:17.215 INFO:teuthology.orchestra.run.smithi188.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-13T03:19:17.217 INFO:teuthology.orchestra.run.smithi188.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-13T03:19:17.220 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-13T03:19:17.221 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-13T03:19:17.313 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout:========================================================================================= 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout:========================================================================================= 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:19.0.0-2884.g218a6914.el9 ceph 6.6 k 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:19.0.0-2884.g218a6914.el9 ceph 5.4 M 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:19.0.0-2884.g218a6914.el9 ceph 891 k 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2884.g218a6914.el9 ceph 145 k 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:19.0.0-2884.g218a6914.el9 ceph 1.5 M 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 142 k 2024-04-13T03:19:17.325 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 3.5 M 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 7.4 M 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 49 k 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:19.0.0-2884.g218a6914.el9 ceph 10 M 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:19.0.0-2884.g218a6914.el9 ceph 49 M 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 264 k 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 763 k 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror x86_64 2:19.0.0-2884.g218a6914.el9 ceph 225 k 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-shell x86_64 2:19.0.0-2884.g218a6914.el9 ceph 27 k 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 18 k 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-13T03:19:17.326 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:19.0.0-2884.g218a6914.el9 ceph 32 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 746 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:19.0.0-2884.g218a6914.el9 ceph 126 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:19.0.0-2884.g218a6914.el9 ceph 162 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:19.0.0-2884.g218a6914.el9 ceph 321 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:19.0.0-2884.g218a6914.el9 ceph 298 k 2024-04-13T03:19:17.327 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:19.0.0-2884.g218a6914.el9 ceph 100 k 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:19.0.0-2884.g218a6914.el9 ceph 85 k 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:19.0.0-2884.g218a6914.el9 ceph 3.1 M 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:19.0.0-2884.g218a6914.el9 ceph 170 k 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager x86_64 1:1.47.4-1.el9 baseos 2.3 M 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 baseos 1.8 M 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 baseos 42 k 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 baseos 249 k 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-13T03:19:17.328 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 3.4 M 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 3.1 M 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-13T03:19:17.329 INFO:teuthology.orchestra.run.smithi188.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-13T03:19:17.330 INFO:teuthology.orchestra.run.smithi188.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-04-13T03:19:17.330 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-13T03:19:17.330 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-13T03:19:17.330 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-13T03:19:17.330 INFO:teuthology.orchestra.run.smithi188.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-13T03:19:17.330 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-13T03:19:17.330 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:19.0.0-2884.g218a6914.el9 ceph 21 M 2024-04-13T03:19:17.330 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 34 k 2024-04-13T03:19:17.330 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:19.0.0-2884.g218a6914.el9 ceph 2.4 M 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 249 k 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:19.0.0-2884.g218a6914.el9 ceph 4.7 M 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:19.0.0-2884.g218a6914.el9 ceph 17 M 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 16 k 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:19.0.0-2884.g218a6914.el9 ceph 25 k 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-13T03:19:17.331 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:19.0.0-2884.g218a6914.el9 ceph 163 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 500 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 5.2 M 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-13T03:19:17.332 INFO:teuthology.orchestra.run.smithi188.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-13T03:19:17.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:19.0.0-2884.g218a6914.el9 ceph 45 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:19.0.0-2884.g218a6914.el9 ceph 135 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-13T03:19:17.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-13T03:19:17.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-13T03:19:17.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-13T03:19:17.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-13T03:19:17.338 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:19:17.339 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-13T03:19:17.339 INFO:teuthology.orchestra.run.smithi188.stdout:========================================================================================= 2024-04-13T03:19:17.339 INFO:teuthology.orchestra.run.smithi188.stdout:Install 150 Packages 2024-04-13T03:19:17.339 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 24 Packages 2024-04-13T03:19:17.339 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:19:17.339 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 229 M 2024-04-13T03:19:17.339 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-13T03:19:18.176 INFO:teuthology.orchestra.run.smithi188.stdout:(1/174): ceph-19.0.0-2884.g218a6914.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-13T03:19:18.377 INFO:teuthology.orchestra.run.smithi188.stdout:(2/174): ceph-fuse-19.0.0-2884.g218a6914.el9.x8 4.3 MB/s | 891 kB 00:00 2024-04-13T03:19:18.435 INFO:teuthology.orchestra.run.smithi188.stdout:(3/174): ceph-immutable-object-cache-19.0.0-288 2.4 MB/s | 145 kB 00:00 2024-04-13T03:19:18.685 INFO:teuthology.orchestra.run.smithi188.stdout:(4/174): ceph-mds-19.0.0-2884.g218a6914.el9.x86 9.5 MB/s | 2.4 MB 00:00 2024-04-13T03:19:18.869 INFO:teuthology.orchestra.run.smithi188.stdout:(5/174): ceph-mgr-19.0.0-2884.g218a6914.el9.x86 8.2 MB/s | 1.5 MB 00:00 2024-04-13T03:19:19.411 INFO:teuthology.orchestra.run.smithi188.stdout:(6/174): ceph-mon-19.0.0-2884.g218a6914.el9.x86 8.7 MB/s | 4.7 MB 00:00 2024-04-13T03:19:19.703 INFO:teuthology.orchestra.run.smithi188.stdout:(7/174): ceph-common-19.0.0-2884.g218a6914.el9. 12 MB/s | 21 MB 00:01 2024-04-13T03:19:19.786 INFO:teuthology.orchestra.run.smithi188.stdout:(8/174): ceph-base-19.0.0-2884.g218a6914.el9.x8 3.0 MB/s | 5.4 MB 00:01 2024-04-13T03:19:19.870 INFO:teuthology.orchestra.run.smithi188.stdout:(9/174): ceph-selinux-19.0.0-2884.g218a6914.el9 302 kB/s | 25 kB 00:00 2024-04-13T03:19:20.537 INFO:teuthology.orchestra.run.smithi188.stdout:(10/174): ceph-radosgw-19.0.0-2884.g218a6914.el 12 MB/s | 10 MB 00:00 2024-04-13T03:19:20.612 INFO:teuthology.orchestra.run.smithi188.stdout:(11/174): cephfs-mirror-19.0.0-2884.g218a6914.e 2.9 MB/s | 225 kB 00:00 2024-04-13T03:19:20.670 INFO:teuthology.orchestra.run.smithi188.stdout:(12/174): cephfs-shell-19.0.0-2884.g218a6914.el 466 kB/s | 27 kB 00:00 2024-04-13T03:19:20.721 INFO:teuthology.orchestra.run.smithi188.stdout:(13/174): libcephfs-devel-19.0.0-2884.g218a6914 634 kB/s | 32 kB 00:00 2024-04-13T03:19:20.813 INFO:teuthology.orchestra.run.smithi188.stdout:(14/174): libcephfs2-19.0.0-2884.g218a6914.el9. 7.9 MB/s | 746 kB 00:00 2024-04-13T03:19:20.888 INFO:teuthology.orchestra.run.smithi188.stdout:(15/174): libcephsqlite-19.0.0-2884.g218a6914.e 2.1 MB/s | 163 kB 00:00 2024-04-13T03:19:21.055 INFO:teuthology.orchestra.run.smithi188.stdout:(16/174): ceph-osd-19.0.0-2884.g218a6914.el9.x8 10 MB/s | 17 MB 00:01 2024-04-13T03:19:21.080 INFO:teuthology.orchestra.run.smithi188.stdout:(17/174): librados-devel-19.0.0-2884.g218a6914. 660 kB/s | 126 kB 00:00 2024-04-13T03:19:21.131 INFO:teuthology.orchestra.run.smithi188.stdout:(18/174): libradosstriper1-19.0.0-2884.g218a691 6.5 MB/s | 500 kB 00:00 2024-04-13T03:19:21.222 INFO:teuthology.orchestra.run.smithi188.stdout:(19/174): python3-ceph-argparse-19.0.0-2884.g21 494 kB/s | 45 kB 00:00 2024-04-13T03:19:21.323 INFO:teuthology.orchestra.run.smithi188.stdout:(20/174): python3-ceph-common-19.0.0-2884.g218a 1.3 MB/s | 135 kB 00:00 2024-04-13T03:19:21.423 INFO:teuthology.orchestra.run.smithi188.stdout:(21/174): python3-cephfs-19.0.0-2884.g218a6914. 1.6 MB/s | 162 kB 00:00 2024-04-13T03:19:21.523 INFO:teuthology.orchestra.run.smithi188.stdout:(22/174): librgw2-19.0.0-2884.g218a6914.el9.x86 12 MB/s | 5.2 MB 00:00 2024-04-13T03:19:21.548 INFO:teuthology.orchestra.run.smithi188.stdout:(23/174): python3-rados-19.0.0-2884.g218a6914.e 2.5 MB/s | 321 kB 00:00 2024-04-13T03:19:21.582 INFO:teuthology.orchestra.run.smithi188.stdout:(24/174): python3-rbd-19.0.0-2884.g218a6914.el9 5.0 MB/s | 298 kB 00:00 2024-04-13T03:19:21.607 INFO:teuthology.orchestra.run.smithi188.stdout:(25/174): python3-rgw-19.0.0-2884.g218a6914.el9 1.7 MB/s | 100 kB 00:00 2024-04-13T03:19:21.632 INFO:teuthology.orchestra.run.smithi188.stdout:(26/174): rbd-fuse-19.0.0-2884.g218a6914.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-13T03:19:21.699 INFO:teuthology.orchestra.run.smithi188.stdout:(27/174): rbd-nbd-19.0.0-2884.g218a6914.el9.x86 2.5 MB/s | 170 kB 00:00 2024-04-13T03:19:21.758 INFO:teuthology.orchestra.run.smithi188.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2884.g 585 kB/s | 34 kB 00:00 2024-04-13T03:19:21.833 INFO:teuthology.orchestra.run.smithi188.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2884.g218a691 1.9 MB/s | 142 kB 00:00 2024-04-13T03:19:21.925 INFO:teuthology.orchestra.run.smithi188.stdout:(30/174): rbd-mirror-19.0.0-2884.g218a6914.el9. 9.7 MB/s | 3.1 MB 00:00 2024-04-13T03:19:22.300 INFO:teuthology.orchestra.run.smithi188.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2884.g218a6 7.6 MB/s | 3.5 MB 00:00 2024-04-13T03:19:22.308 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 908 kB/s | 19 MB 00:21 2024-04-13T03:19:22.367 INFO:teuthology.orchestra.run.smithi188.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2884.g21 3.7 MB/s | 249 kB 00:00 2024-04-13T03:19:22.434 INFO:teuthology.orchestra.run.smithi188.stdout:(33/174): ceph-mgr-rook-19.0.0-2884.g218a6914.e 736 kB/s | 49 kB 00:00 2024-04-13T03:19:22.493 INFO:teuthology.orchestra.run.smithi188.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2884.g2 281 kB/s | 16 kB 00:00 2024-04-13T03:19:22.577 INFO:teuthology.orchestra.run.smithi188.stdout:(35/174): ceph-volume-19.0.0-2884.g218a6914.el9 3.1 MB/s | 264 kB 00:00 2024-04-13T03:19:22.777 INFO:teuthology.orchestra.run.smithi188.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 8.7 MB/s | 7.4 MB 00:00 2024-04-13T03:19:22.802 INFO:teuthology.orchestra.run.smithi188.stdout:(37/174): cephadm-19.0.0-2884.g218a6914.el9.noa 3.3 MB/s | 763 kB 00:00 2024-04-13T03:19:22.828 INFO:teuthology.orchestra.run.smithi188.stdout:(38/174): cephfs-top-19.0.0-2884.g218a6914.el9. 364 kB/s | 18 kB 00:00 2024-04-13T03:19:22.894 INFO:teuthology.orchestra.run.smithi188.stdout:(39/174): libaio-devel-0.3.112-1.el8.x86_64.rpm 191 kB/s | 18 kB 00:00 2024-04-13T03:19:23.020 INFO:teuthology.orchestra.run.smithi188.stdout:(40/174): fuse-2.9.9-15.el9.x86_64.rpm 479 kB/s | 80 kB 00:00 2024-04-13T03:19:23.095 INFO:teuthology.orchestra.run.smithi188.stdout:(41/174): jq-1.6-16.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2024-04-13T03:19:23.187 INFO:teuthology.orchestra.run.smithi188.stdout:(42/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 803 kB 00:00 2024-04-13T03:19:23.212 INFO:teuthology.orchestra.run.smithi188.stdout:(43/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-04-13T03:19:23.245 INFO:teuthology.orchestra.run.smithi188.stdout:(44/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 218 kB 00:00 2024-04-13T03:19:23.271 INFO:teuthology.orchestra.run.smithi188.stdout:(45/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-04-13T03:19:23.338 INFO:teuthology.orchestra.run.smithi188.stdout:(46/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-13T03:19:23.374 INFO:teuthology.orchestra.run.smithi188.stdout:(47/174): python3-cryptography-36.0.1-4.el9.x86 9.6 MB/s | 1.2 MB 00:00 2024-04-13T03:19:23.399 INFO:teuthology.orchestra.run.smithi188.stdout:(48/174): python3-pycparser-2.20-6.el9.noarch.r 2.2 MB/s | 135 kB 00:00 2024-04-13T03:19:23.833 INFO:teuthology.orchestra.run.smithi188.stdout:(49/174): ceph-test-19.0.0-2884.g218a6914.el9.x 12 MB/s | 49 MB 00:03 2024-04-13T03:19:23.858 INFO:teuthology.orchestra.run.smithi188.stdout:(50/174): python3-requests-2.25.1-8.el9.noarch. 258 kB/s | 125 kB 00:00 2024-04-13T03:19:23.884 INFO:teuthology.orchestra.run.smithi188.stdout:(51/174): python3-urllib3-1.26.5-5.el9.noarch.r 443 kB/s | 215 kB 00:00 2024-04-13T03:19:24.114 INFO:teuthology.orchestra.run.smithi188.stdout:(52/174): boost-program-options-1.75.0-8.el9.x8 465 kB/s | 107 kB 00:00 2024-04-13T03:19:24.176 INFO:teuthology.orchestra.run.smithi188.stdout:(53/174): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 689 kB 00:00 2024-04-13T03:19:24.193 INFO:teuthology.orchestra.run.smithi188.stdout:(54/174): elfutils-libelf-devel-0.190-2.el9.x86 273 kB/s | 21 kB 00:00 2024-04-13T03:19:24.235 INFO:teuthology.orchestra.run.smithi188.stdout:(55/174): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 943 kB 00:00 2024-04-13T03:19:24.260 INFO:teuthology.orchestra.run.smithi188.stdout:(56/174): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 313 kB 00:00 2024-04-13T03:19:24.277 INFO:teuthology.orchestra.run.smithi188.stdout:(57/174): flexiblas-3.0.4-8.el9.x86_64.rpm 391 kB/s | 33 kB 00:00 2024-04-13T03:19:24.282 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 892 kB/s | 19 MB 00:21 2024-04-13T03:19:24.311 INFO:teuthology.orchestra.run.smithi188.stdout:(58/174): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-04-13T03:19:24.344 INFO:teuthology.orchestra.run.smithi188.stdout:(59/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-04-13T03:19:24.394 INFO:teuthology.orchestra.run.smithi188.stdout:(60/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-04-13T03:19:24.411 INFO:teuthology.orchestra.run.smithi188.stdout:(61/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 450 kB/s | 45 kB 00:00 2024-04-13T03:19:24.445 INFO:teuthology.orchestra.run.smithi188.stdout:(62/174): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-04-13T03:19:24.503 INFO:teuthology.orchestra.run.smithi188.stdout:(63/174): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 582 kB 00:00 2024-04-13T03:19:24.520 INFO:teuthology.orchestra.run.smithi188.stdout:(64/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 233 kB/s | 25 kB 00:00 2024-04-13T03:19:24.537 INFO:teuthology.orchestra.run.smithi188.stdout:(65/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 514 kB/s | 47 kB 00:00 2024-04-13T03:19:24.588 INFO:teuthology.orchestra.run.smithi188.stdout:(66/174): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-13T03:19:24.604 INFO:teuthology.orchestra.run.smithi188.stdout:(67/174): openblas-0.3.21-2.el9.x86_64.rpm 527 kB/s | 35 kB 00:00 2024-04-13T03:19:24.630 INFO:teuthology.orchestra.run.smithi188.stdout:(68/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-04-13T03:19:24.797 INFO:teuthology.orchestra.run.smithi188.stdout:(69/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 22 MB/s | 4.1 MB 00:00 2024-04-13T03:19:24.864 INFO:teuthology.orchestra.run.smithi188.stdout:(70/174): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-04-13T03:19:24.905 INFO:teuthology.orchestra.run.smithi188.stdout:(71/174): postgresql-13.14-1.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-04-13T03:19:24.947 INFO:teuthology.orchestra.run.smithi188.stdout:(72/174): postgresql-contrib-13.14-1.el9.x86_64 5.9 MB/s | 911 kB 00:00 2024-04-13T03:19:24.973 INFO:teuthology.orchestra.run.smithi188.stdout:(73/174): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-13T03:19:25.065 INFO:teuthology.orchestra.run.smithi188.stdout:(74/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-04-13T03:19:25.090 INFO:teuthology.orchestra.run.smithi188.stdout:(75/174): python3-attrs-20.3.0-7.el9.noarch.rpm 749 kB/s | 87 kB 00:00 2024-04-13T03:19:25.207 INFO:teuthology.orchestra.run.smithi188.stdout:(76/174): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.8 MB 00:00 2024-04-13T03:19:25.257 INFO:teuthology.orchestra.run.smithi188.stdout:(77/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-04-13T03:19:25.282 INFO:teuthology.orchestra.run.smithi188.stdout:(78/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-04-13T03:19:25.374 INFO:teuthology.orchestra.run.smithi188.stdout:(79/174): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-04-13T03:19:25.400 INFO:teuthology.orchestra.run.smithi188.stdout:(80/174): python3-jsonpatch-1.21-16.el9.noarch. 181 kB/s | 26 kB 00:00 2024-04-13T03:19:25.425 INFO:teuthology.orchestra.run.smithi188.stdout:(81/174): python3-jsonpointer-2.0-4.el9.noarch. 135 kB/s | 19 kB 00:00 2024-04-13T03:19:25.450 INFO:teuthology.orchestra.run.smithi188.stdout:(82/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-13T03:19:25.475 INFO:teuthology.orchestra.run.smithi188.stdout:(83/174): python3-markupsafe-1.1.1-12.el9.x86_6 461 kB/s | 35 kB 00:00 2024-04-13T03:19:25.526 INFO:teuthology.orchestra.run.smithi188.stdout:(84/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-13T03:19:25.551 INFO:teuthology.orchestra.run.smithi188.stdout:(85/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-04-13T03:19:25.626 INFO:teuthology.orchestra.run.smithi188.stdout:(86/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-04-13T03:19:25.652 INFO:teuthology.orchestra.run.smithi188.stdout:(87/174): python3-packaging-20.9-5.el9.noarch.r 616 kB/s | 77 kB 00:00 2024-04-13T03:19:25.677 INFO:teuthology.orchestra.run.smithi188.stdout:(88/174): python3-prettytable-0.7.2-27.el9.noar 337 kB/s | 42 kB 00:00 2024-04-13T03:19:25.702 INFO:teuthology.orchestra.run.smithi188.stdout:(89/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-13T03:19:25.727 INFO:teuthology.orchestra.run.smithi188.stdout:(90/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-13T03:19:25.752 INFO:teuthology.orchestra.run.smithi188.stdout:(91/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-13T03:19:25.769 INFO:teuthology.orchestra.run.smithi188.stdout:(92/174): python3-pytz-2021.1-5.el9.noarch.rpm 762 kB/s | 51 kB 00:00 2024-04-13T03:19:25.786 INFO:teuthology.orchestra.run.smithi188.stdout:(93/174): python3-requests-oauthlib-1.3.0-12.el 910 kB/s | 54 kB 00:00 2024-04-13T03:19:25.836 INFO:teuthology.orchestra.run.smithi188.stdout:(94/174): python3-toml-0.10.2-6.el9.noarch.rpm 622 kB/s | 42 kB 00:00 2024-04-13T03:19:25.861 INFO:teuthology.orchestra.run.smithi188.stdout:(95/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 589 kB/s | 44 kB 00:00 2024-04-13T03:19:25.920 INFO:teuthology.orchestra.run.smithi188.stdout:(96/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-04-13T03:19:25.946 INFO:teuthology.orchestra.run.smithi188.stdout:(97/174): qatlib-service-24.02.0-1.el9.x86_64.r 427 kB/s | 36 kB 00:00 2024-04-13T03:19:25.980 INFO:teuthology.orchestra.run.smithi188.stdout:(98/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 807 kB/s | 47 kB 00:00 2024-04-13T03:19:26.038 INFO:teuthology.orchestra.run.smithi188.stdout:(99/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 305 kB 00:00 2024-04-13T03:19:26.055 INFO:teuthology.orchestra.run.smithi188.stdout:(100/174): uuid-1.6.2-55.el9.x86_64.rpm 769 kB/s | 58 kB 00:00 2024-04-13T03:19:26.222 INFO:teuthology.orchestra.run.smithi188.stdout:(101/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-04-13T03:19:26.249 INFO:teuthology.orchestra.run.smithi188.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 444 kB/s | 94 kB 00:00 2024-04-13T03:19:26.275 INFO:teuthology.orchestra.run.smithi188.stdout:(103/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 290 kB/s | 64 kB 00:00 2024-04-13T03:19:26.300 INFO:teuthology.orchestra.run.smithi188.stdout:(104/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 582 kB/s | 45 kB 00:00 2024-04-13T03:19:26.550 INFO:teuthology.orchestra.run.smithi188.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-04-13T03:19:26.584 INFO:teuthology.orchestra.run.smithi188.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-04-13T03:19:26.592 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.2 MB 00:00 2024-04-13T03:19:26.626 INFO:teuthology.orchestra.run.smithi188.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.0 MB/s | 309 kB 00:00 2024-04-13T03:19:26.668 INFO:teuthology.orchestra.run.smithi188.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 232 kB/s | 19 kB 00:00 2024-04-13T03:19:26.877 INFO:teuthology.orchestra.run.smithi188.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 125 kB/s | 26 kB 00:00 2024-04-13T03:19:26.952 INFO:teuthology.orchestra.run.smithi188.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 669 kB/s | 49 kB 00:00 2024-04-13T03:19:27.027 INFO:teuthology.orchestra.run.smithi188.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-13T03:19:27.052 INFO:teuthology.orchestra.run.smithi188.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-04-13T03:19:27.136 INFO:teuthology.orchestra.run.smithi188.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-04-13T03:19:27.219 INFO:teuthology.orchestra.run.smithi188.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-04-13T03:19:27.278 INFO:teuthology.orchestra.run.smithi188.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-04-13T03:19:27.295 INFO:teuthology.orchestra.run.smithi188.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 581 kB/s | 43 kB 00:00 2024-04-13T03:19:27.320 INFO:teuthology.orchestra.run.smithi188.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 773 kB/s | 32 kB 00:00 2024-04-13T03:19:27.337 INFO:teuthology.orchestra.run.smithi188.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 357 kB/s | 15 kB 00:00 2024-04-13T03:19:27.387 INFO:teuthology.orchestra.run.smithi188.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-13T03:19:27.413 INFO:teuthology.orchestra.run.smithi188.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-13T03:19:27.455 INFO:teuthology.orchestra.run.smithi188.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.5 MB/s | 303 kB 00:00 2024-04-13T03:19:27.480 INFO:teuthology.orchestra.run.smithi188.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 529 kB/s | 35 kB 00:00 2024-04-13T03:19:27.513 INFO:teuthology.orchestra.run.smithi188.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-04-13T03:19:27.639 INFO:teuthology.orchestra.run.smithi188.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-04-13T03:19:27.664 INFO:teuthology.orchestra.run.smithi188.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 960 kB/s | 144 kB 00:00 2024-04-13T03:19:27.698 INFO:teuthology.orchestra.run.smithi188.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-13T03:19:27.723 INFO:teuthology.orchestra.run.smithi188.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 304 kB/s | 18 kB 00:00 2024-04-13T03:19:27.748 INFO:teuthology.orchestra.run.smithi188.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-04-13T03:19:27.774 INFO:teuthology.orchestra.run.smithi188.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-04-13T03:19:27.849 INFO:teuthology.orchestra.run.smithi188.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-13T03:19:27.891 INFO:teuthology.orchestra.run.smithi188.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-13T03:19:27.916 INFO:teuthology.orchestra.run.smithi188.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 713 kB/s | 46 kB 00:00 2024-04-13T03:19:27.941 INFO:teuthology.orchestra.run.smithi188.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-13T03:19:27.975 INFO:teuthology.orchestra.run.smithi188.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 986 kB/s | 58 kB 00:00 2024-04-13T03:19:28.009 INFO:teuthology.orchestra.run.smithi188.stdout:(135/174): protobuf-compiler-3.14.0-13.el9.x86_ 498 kB/s | 863 kB 00:01 2024-04-13T03:19:28.051 INFO:teuthology.orchestra.run.smithi188.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-13T03:19:28.076 INFO:teuthology.orchestra.run.smithi188.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-04-13T03:19:28.101 INFO:teuthology.orchestra.run.smithi188.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 976 kB/s | 90 kB 00:00 2024-04-13T03:19:28.126 INFO:teuthology.orchestra.run.smithi188.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-04-13T03:19:28.152 INFO:teuthology.orchestra.run.smithi188.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-04-13T03:19:28.177 INFO:teuthology.orchestra.run.smithi188.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-13T03:19:28.203 INFO:teuthology.orchestra.run.smithi188.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 786 kB/s | 59 kB 00:00 2024-04-13T03:19:28.229 INFO:teuthology.orchestra.run.smithi188.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 476 kB/s | 36 kB 00:00 2024-04-13T03:19:28.253 INFO:teuthology.orchestra.run.smithi188.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 681 kB/s | 51 kB 00:00 2024-04-13T03:19:28.286 INFO:teuthology.orchestra.run.smithi188.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-04-13T03:19:28.311 INFO:teuthology.orchestra.run.smithi188.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-04-13T03:19:28.337 INFO:teuthology.orchestra.run.smithi188.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-13T03:19:28.362 INFO:teuthology.orchestra.run.smithi188.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 269 kB/s | 20 kB 00:00 2024-04-13T03:19:28.396 INFO:teuthology.orchestra.run.smithi188.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-13T03:19:28.529 INFO:teuthology.orchestra.run.smithi188.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 8.3 MB/s | 1.6 MB 00:00 2024-04-13T03:19:28.688 INFO:teuthology.orchestra.run.smithi188.stdout:(151/174): librados2-19.0.0-2884.g218a6914.el9. 10 MB/s | 3.4 MB 00:00 2024-04-13T03:19:28.855 INFO:teuthology.orchestra.run.smithi188.stdout:(152/174): librbd1-19.0.0-2884.g218a6914.el9.x8 6.8 MB/s | 3.1 MB 00:00 2024-04-13T03:19:28.897 INFO:teuthology.orchestra.run.smithi188.stdout:(153/174): NetworkManager-1.47.4-1.el9.x86_64.r 6.3 MB/s | 2.3 MB 00:00 2024-04-13T03:19:28.964 INFO:teuthology.orchestra.run.smithi188.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 3.7 MB/s | 249 kB 00:00 2024-04-13T03:19:29.009 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-04-13T03:19:29.014 INFO:teuthology.orchestra.run.smithi188.stdout:(155/174): NetworkManager-team-1.47.4-1.el9.x86 264 kB/s | 42 kB 00:00 2024-04-13T03:19:29.039 INFO:teuthology.orchestra.run.smithi188.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 948 kB/s | 71 kB 00:00 2024-04-13T03:19:29.081 INFO:teuthology.orchestra.run.smithi188.stdout:(157/174): NetworkManager-libnm-1.47.4-1.el9.x8 4.7 MB/s | 1.8 MB 00:00 2024-04-13T03:19:29.107 INFO:teuthology.orchestra.run.smithi188.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 395 kB/s | 36 kB 00:00 2024-04-13T03:19:29.132 INFO:teuthology.orchestra.run.smithi188.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-04-13T03:19:29.157 INFO:teuthology.orchestra.run.smithi188.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.3 MB/s | 256 kB 00:00 2024-04-13T03:19:29.182 INFO:teuthology.orchestra.run.smithi188.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 304 kB/s | 23 kB 00:00 2024-04-13T03:19:29.233 INFO:teuthology.orchestra.run.smithi188.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-04-13T03:19:29.258 INFO:teuthology.orchestra.run.smithi188.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-04-13T03:19:29.284 INFO:teuthology.orchestra.run.smithi188.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-04-13T03:19:29.309 INFO:teuthology.orchestra.run.smithi188.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 847 kB/s | 64 kB 00:00 2024-04-13T03:19:29.334 INFO:teuthology.orchestra.run.smithi188.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-04-13T03:19:29.409 INFO:teuthology.orchestra.run.smithi188.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 21 MB/s | 2.2 MB 00:00 2024-04-13T03:19:29.460 INFO:teuthology.orchestra.run.smithi188.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2024-04-13T03:19:29.488 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.2 MB 00:01 2024-04-13T03:19:29.502 INFO:teuthology.orchestra.run.smithi188.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-04-13T03:19:29.527 INFO:teuthology.orchestra.run.smithi188.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 466 kB/s | 31 kB 00:00 2024-04-13T03:19:29.552 INFO:teuthology.orchestra.run.smithi188.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-04-13T03:19:29.577 INFO:teuthology.orchestra.run.smithi188.stdout:(172/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2024-04-13T03:19:29.611 INFO:teuthology.orchestra.run.smithi188.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-04-13T03:19:29.661 INFO:teuthology.orchestra.run.smithi188.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 684 kB/s | 74 kB 00:00 2024-04-13T03:19:29.667 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-13T03:19:29.667 INFO:teuthology.orchestra.run.smithi188.stdout:Total 19 MB/s | 229 MB 00:12 2024-04-13T03:19:30.086 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-13T03:19:31.046 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-13T03:19:31.322 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-13T03:19:31.322 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-13T03:19:31.325 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-13T03:19:32.569 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-04-13T03:19:33.171 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-13T03:19:33.172 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-13T03:19:35.220 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-04-13T03:19:37.772 INFO:teuthology.orchestra.run.smithi119.stderr:Modular dependency problems: 2024-04-13T03:19:37.772 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-13T03:19:37.772 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 1: conflicting requests 2024-04-13T03:19:37.772 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 2: conflicting requests 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 3: conflicting requests 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 4: conflicting requests 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 5: conflicting requests 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 6: conflicting requests 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 7: conflicting requests 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.773 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 8: conflicting requests 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 9: conflicting requests 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 10: conflicting requests 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 11: conflicting requests 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 12: conflicting requests 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 13: conflicting requests 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 14: conflicting requests 2024-04-13T03:19:37.774 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 15: conflicting requests 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 16: conflicting requests 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 17: conflicting requests 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 18: conflicting requests 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 19: conflicting requests 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 20: conflicting requests 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.775 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 21: conflicting requests 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 22: conflicting requests 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 23: conflicting requests 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 24: conflicting requests 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 25: conflicting requests 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 26: conflicting requests 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 27: conflicting requests 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.776 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 28: conflicting requests 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 29: conflicting requests 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 30: conflicting requests 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 31: conflicting requests 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 32: conflicting requests 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 33: conflicting requests 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 34: conflicting requests 2024-04-13T03:19:37.777 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 35: conflicting requests 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 36: conflicting requests 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 37: conflicting requests 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.778 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 39: conflicting requests 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.779 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.780 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T03:19:37.781 INFO:teuthology.orchestra.run.smithi119.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T03:19:38.199 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-13T03:19:38.200 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-13T03:19:38.213 INFO:teuthology.orchestra.run.smithi119.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-13T03:19:38.214 INFO:teuthology.orchestra.run.smithi119.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-13T03:19:38.215 INFO:teuthology.orchestra.run.smithi119.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-13T03:19:38.216 INFO:teuthology.orchestra.run.smithi119.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-13T03:19:38.218 INFO:teuthology.orchestra.run.smithi119.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-13T03:19:38.221 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-13T03:19:38.222 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-13T03:19:38.313 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-13T03:19:38.331 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-04-13T03:19:38.331 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-13T03:19:38.331 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-04-13T03:19:38.331 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-13T03:19:38.331 INFO:teuthology.orchestra.run.smithi119.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-13T03:19:38.331 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.0.0-2884.g218a6914.el9 ceph 6.6 k 2024-04-13T03:19:38.331 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.0.0-2884.g218a6914.el9 ceph 5.4 M 2024-04-13T03:19:38.331 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.0.0-2884.g218a6914.el9 ceph 891 k 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2884.g218a6914.el9 ceph 145 k 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.0.0-2884.g218a6914.el9 ceph 1.5 M 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 142 k 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 3.5 M 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 7.4 M 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 49 k 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.0.0-2884.g218a6914.el9 ceph 10 M 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.0.0-2884.g218a6914.el9 ceph 49 M 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 264 k 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 763 k 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror x86_64 2:19.0.0-2884.g218a6914.el9 ceph 225 k 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell x86_64 2:19.0.0-2884.g218a6914.el9 ceph 27 k 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 18 k 2024-04-13T03:19:38.332 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.0.0-2884.g218a6914.el9 ceph 32 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 746 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.0.0-2884.g218a6914.el9 ceph 126 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.0.0-2884.g218a6914.el9 ceph 162 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.0.0-2884.g218a6914.el9 ceph 321 k 2024-04-13T03:19:38.333 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.0.0-2884.g218a6914.el9 ceph 298 k 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.0.0-2884.g218a6914.el9 ceph 100 k 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.0.0-2884.g218a6914.el9 ceph 85 k 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.0.0-2884.g218a6914.el9 ceph 3.1 M 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.0.0-2884.g218a6914.el9 ceph 170 k 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager x86_64 1:1.47.4-1.el9 baseos 2.3 M 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 baseos 1.8 M 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 baseos 42 k 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 baseos 249 k 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-13T03:19:38.334 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 3.4 M 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 3.1 M 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-13T03:19:38.335 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.0.0-2884.g218a6914.el9 ceph 21 M 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 34 k 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.0.0-2884.g218a6914.el9 ceph 2.4 M 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 249 k 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.0.0-2884.g218a6914.el9 ceph 4.7 M 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.0.0-2884.g218a6914.el9 ceph 17 M 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 16 k 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.0.0-2884.g218a6914.el9 ceph 25 k 2024-04-13T03:19:38.336 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.0.0-2884.g218a6914.el9 ceph 163 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-13T03:19:38.337 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 500 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 5.2 M 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-13T03:19:38.338 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:19.0.0-2884.g218a6914.el9 ceph 45 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.0.0-2884.g218a6914.el9 ceph 135 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-13T03:19:38.339 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-13T03:19:38.340 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-13T03:19:38.341 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-13T03:19:38.342 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-13T03:19:38.343 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-13T03:19:38.344 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-04-13T03:19:38.344 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-04-13T03:19:38.344 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-13T03:19:38.344 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:19:38.344 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-13T03:19:38.344 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-04-13T03:19:38.344 INFO:teuthology.orchestra.run.smithi119.stdout:Install 150 Packages 2024-04-13T03:19:38.344 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 24 Packages 2024-04-13T03:19:38.344 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:19:38.345 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 229 M 2024-04-13T03:19:38.345 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-13T03:19:38.835 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-13T03:19:39.187 INFO:teuthology.orchestra.run.smithi119.stdout:(1/174): ceph-19.0.0-2884.g218a6914.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-13T03:19:39.371 INFO:teuthology.orchestra.run.smithi119.stdout:(2/174): ceph-fuse-19.0.0-2884.g218a6914.el9.x8 4.7 MB/s | 891 kB 00:00 2024-04-13T03:19:39.421 INFO:teuthology.orchestra.run.smithi119.stdout:(3/174): ceph-immutable-object-cache-19.0.0-288 2.8 MB/s | 145 kB 00:00 2024-04-13T03:19:39.646 INFO:teuthology.orchestra.run.smithi119.stdout:(4/174): ceph-mds-19.0.0-2884.g218a6914.el9.x86 11 MB/s | 2.4 MB 00:00 2024-04-13T03:19:39.888 INFO:teuthology.orchestra.run.smithi119.stdout:(5/174): ceph-mgr-19.0.0-2884.g218a6914.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-04-13T03:19:39.972 INFO:teuthology.orchestra.run.smithi119.stdout:(6/174): ceph-base-19.0.0-2884.g218a6914.el9.x8 5.6 MB/s | 5.4 MB 00:00 2024-04-13T03:19:40.223 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-13T03:19:40.397 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-13T03:19:40.431 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-13T03:19:40.462 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-13T03:19:40.472 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-13T03:19:40.512 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-13T03:19:40.574 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-13T03:19:40.607 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-13T03:19:40.633 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-13T03:19:40.675 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-13T03:19:40.813 INFO:teuthology.orchestra.run.smithi119.stdout:(7/174): ceph-mon-19.0.0-2884.g218a6914.el9.x86 5.1 MB/s | 4.7 MB 00:00 2024-04-13T03:19:40.995 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-13T03:19:41.051 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:19.0.0-2884.g218a6914.el9.x86_64 10/198 2024-04-13T03:19:41.171 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:19.0.0-2884.g218a6914.el9.x86_64 10/198 2024-04-13T03:19:41.202 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 11/198 2024-04-13T03:19:41.275 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 11/198 2024-04-13T03:19:41.306 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:19.0.0-2884.g218a6914.el9.x86_64 12/198 2024-04-13T03:19:41.354 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-13T03:19:41.364 INFO:teuthology.orchestra.run.smithi119.stdout:(8/174): ceph-common-19.0.0-2884.g218a6914.el9. 8.9 MB/s | 21 MB 00:02 2024-04-13T03:19:41.431 INFO:teuthology.orchestra.run.smithi119.stdout:(9/174): ceph-selinux-19.0.0-2884.g218a6914.el9 379 kB/s | 25 kB 00:00 2024-04-13T03:19:41.512 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-13T03:19:41.562 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-13T03:19:41.590 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 16/198 2024-04-13T03:19:41.619 INFO:teuthology.orchestra.run.smithi016.stderr:Modular dependency problems: 2024-04-13T03:19:41.619 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-13T03:19:41.619 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 1: conflicting requests 2024-04-13T03:19:41.619 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.619 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 2: conflicting requests 2024-04-13T03:19:41.619 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.619 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 3: conflicting requests 2024-04-13T03:19:41.619 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 4: conflicting requests 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 5: conflicting requests 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 6: conflicting requests 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 7: conflicting requests 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 8: conflicting requests 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 9: conflicting requests 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.620 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 10: conflicting requests 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 11: conflicting requests 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 12: conflicting requests 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 13: conflicting requests 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 14: conflicting requests 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 15: conflicting requests 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.621 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 16: conflicting requests 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 17: conflicting requests 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 18: conflicting requests 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 19: conflicting requests 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 20: conflicting requests 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 21: conflicting requests 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 22: conflicting requests 2024-04-13T03:19:41.622 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 23: conflicting requests 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 24: conflicting requests 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 25: conflicting requests 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 26: conflicting requests 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 27: conflicting requests 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 28: conflicting requests 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 29: conflicting requests 2024-04-13T03:19:41.623 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 30: conflicting requests 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 31: conflicting requests 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 32: conflicting requests 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 33: conflicting requests 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 34: conflicting requests 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 35: conflicting requests 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.624 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 36: conflicting requests 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 37: conflicting requests 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 38: cannot install the best candidate for the job 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 39: conflicting requests 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.625 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 40: cannot install the best candidate for the job 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 41: cannot install the best candidate for the job 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.626 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 42: cannot install the best candidate for the job 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 43: cannot install the best candidate for the job 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 44: cannot install the best candidate for the job 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.627 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.628 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.628 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 45: cannot install the best candidate for the job 2024-04-13T03:19:41.628 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.628 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.628 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.628 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-13T03:19:41.692 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 16/198 2024-04-13T03:19:41.717 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86 17/198 2024-04-13T03:19:41.780 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86 17/198 2024-04-13T03:19:41.820 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-13T03:19:41.843 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-13T03:19:41.859 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-13T03:19:41.881 INFO:teuthology.orchestra.run.smithi119.stdout:(10/174): ceph-osd-19.0.0-2884.g218a6914.el9.x8 8.8 MB/s | 17 MB 00:01 2024-04-13T03:19:41.973 INFO:teuthology.orchestra.run.smithi119.stdout:(11/174): cephfs-mirror-19.0.0-2884.g218a6914.e 2.4 MB/s | 225 kB 00:00 2024-04-13T03:19:42.051 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-13T03:19:42.082 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-13T03:19:42.114 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-13T03:19:42.115 INFO:teuthology.orchestra.run.smithi119.stdout:(12/174): ceph-radosgw-19.0.0-2884.g218a6914.el 7.7 MB/s | 10 MB 00:01 2024-04-13T03:19:42.132 INFO:teuthology.orchestra.run.smithi119.stdout:(13/174): cephfs-shell-19.0.0-2884.g218a6914.el 170 kB/s | 27 kB 00:00 2024-04-13T03:19:42.166 INFO:teuthology.orchestra.run.smithi119.stdout:(14/174): libcephfs-devel-19.0.0-2884.g218a6914 634 kB/s | 32 kB 00:00 2024-04-13T03:19:42.209 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-13T03:19:42.224 INFO:teuthology.orchestra.run.smithi119.stdout:(15/174): libcephsqlite-19.0.0-2884.g218a6914.e 2.7 MB/s | 163 kB 00:00 2024-04-13T03:19:42.250 INFO:teuthology.orchestra.run.smithi119.stdout:(16/174): libcephfs2-19.0.0-2884.g218a6914.el9. 6.2 MB/s | 746 kB 00:00 2024-04-13T03:19:42.283 INFO:teuthology.orchestra.run.smithi119.stdout:(17/174): librados-devel-19.0.0-2884.g218a6914. 2.1 MB/s | 126 kB 00:00 2024-04-13T03:19:42.294 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-13T03:19:42.296 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-13T03:19:42.308 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-13T03:19:42.318 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-13T03:19:42.320 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-13T03:19:42.321 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-13T03:19:42.323 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-13T03:19:42.325 INFO:teuthology.orchestra.run.smithi119.stdout:(18/174): libradosstriper1-19.0.0-2884.g218a691 6.5 MB/s | 500 kB 00:00 2024-04-13T03:19:42.325 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-13T03:19:42.329 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-13T03:19:42.331 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-13T03:19:42.370 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-13T03:19:42.409 INFO:teuthology.orchestra.run.smithi119.stdout:(19/174): python3-ceph-argparse-19.0.0-2884.g21 541 kB/s | 45 kB 00:00 2024-04-13T03:19:42.440 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-13T03:19:42.440 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================= 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================= 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.0.0-2884.g218a6914.el9 ceph 6.6 k 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.0.0-2884.g218a6914.el9 ceph 5.4 M 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.0.0-2884.g218a6914.el9 ceph 891 k 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2884.g218a6914.el9 ceph 145 k 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.0.0-2884.g218a6914.el9 ceph 1.5 M 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 142 k 2024-04-13T03:19:42.460 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 3.5 M 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 7.4 M 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 49 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.0.0-2884.g218a6914.el9 ceph 10 M 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.0.0-2884.g218a6914.el9 ceph 49 M 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 264 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 763 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:19.0.0-2884.g218a6914.el9 ceph 225 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell x86_64 2:19.0.0-2884.g218a6914.el9 ceph 27 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 18 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.0.0-2884.g218a6914.el9 ceph 32 k 2024-04-13T03:19:42.461 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 746 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.0.0-2884.g218a6914.el9 ceph 126 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.0.0-2884.g218a6914.el9 ceph 162 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.0.0-2884.g218a6914.el9 ceph 321 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.0.0-2884.g218a6914.el9 ceph 298 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.0.0-2884.g218a6914.el9 ceph 100 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.0.0-2884.g218a6914.el9 ceph 85 k 2024-04-13T03:19:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.0.0-2884.g218a6914.el9 ceph 3.1 M 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.0.0-2884.g218a6914.el9 ceph 170 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager x86_64 1:1.47.4-1.el9 baseos 2.3 M 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 baseos 1.8 M 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 baseos 42 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 baseos 249 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-13T03:19:42.463 INFO:teuthology.orchestra.run.smithi016.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 3.4 M 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 3.1 M 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-04-13T03:19:42.464 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.0.0-2884.g218a6914.el9 ceph 21 M 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 34 k 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.0.0-2884.g218a6914.el9 ceph 2.4 M 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 249 k 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.0.0-2884.g218a6914.el9 ceph 4.7 M 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.0.0-2884.g218a6914.el9 ceph 17 M 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2884.g218a6914.el9 ceph-noarch 16 k 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.0.0-2884.g218a6914.el9 ceph 25 k 2024-04-13T03:19:42.465 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.0.0-2884.g218a6914.el9 ceph 163 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-13T03:19:42.466 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 500 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.0.0-2884.g218a6914.el9 ceph 5.2 M 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-13T03:19:42.467 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.0.0-2884.g218a6914.el9 ceph 45 k 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.0.0-2884.g218a6914.el9 ceph 135 k 2024-04-13T03:19:42.468 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-13T03:19:42.469 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-13T03:19:42.470 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-13T03:19:42.470 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-13T03:19:42.470 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-13T03:19:42.470 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-13T03:19:42.470 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-13T03:19:42.470 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-13T03:19:42.470 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-13T03:19:42.470 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-13T03:19:42.470 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-13T03:19:42.471 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-13T03:19:42.472 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-13T03:19:42.473 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================= 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout:Install 150 Packages 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 24 Packages 2024-04-13T03:19:42.474 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:19:42.475 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 229 M 2024-04-13T03:19:42.475 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-13T03:19:42.484 INFO:teuthology.orchestra.run.smithi119.stdout:(20/174): python3-ceph-common-19.0.0-2884.g218a 1.8 MB/s | 135 kB 00:00 2024-04-13T03:19:42.511 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-13T03:19:42.559 INFO:teuthology.orchestra.run.smithi119.stdout:(21/174): python3-cephfs-19.0.0-2884.g218a6914. 2.1 MB/s | 162 kB 00:00 2024-04-13T03:19:42.643 INFO:teuthology.orchestra.run.smithi119.stdout:(22/174): python3-rados-19.0.0-2884.g218a6914.e 3.8 MB/s | 321 kB 00:00 2024-04-13T03:19:42.726 INFO:teuthology.orchestra.run.smithi119.stdout:(23/174): python3-rbd-19.0.0-2884.g218a6914.el9 3.5 MB/s | 298 kB 00:00 2024-04-13T03:19:42.737 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-13T03:19:42.827 INFO:teuthology.orchestra.run.smithi119.stdout:(24/174): librgw2-19.0.0-2884.g218a6914.el9.x86 9.6 MB/s | 5.2 MB 00:00 2024-04-13T03:19:42.852 INFO:teuthology.orchestra.run.smithi119.stdout:(25/174): python3-rgw-19.0.0-2884.g218a6914.el9 795 kB/s | 100 kB 00:00 2024-04-13T03:19:42.858 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-13T03:19:42.877 INFO:teuthology.orchestra.run.smithi119.stdout:(26/174): rbd-fuse-19.0.0-2884.g218a6914.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-13T03:19:42.963 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-13T03:19:42.965 INFO:teuthology.orchestra.run.smithi119.stdout:(27/174): rbd-nbd-19.0.0-2884.g218a6914.el9.x86 1.9 MB/s | 170 kB 00:00 2024-04-13T03:19:43.002 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-13T03:19:43.036 INFO:teuthology.orchestra.run.smithi119.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2884.g 484 kB/s | 34 kB 00:00 2024-04-13T03:19:43.051 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-13T03:19:43.103 INFO:teuthology.orchestra.run.smithi119.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2884.g218a691 2.1 MB/s | 142 kB 00:00 2024-04-13T03:19:43.139 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/198 2024-04-13T03:19:43.175 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/198 2024-04-13T03:19:43.227 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/198 2024-04-13T03:19:43.228 INFO:teuthology.orchestra.run.smithi119.stdout:(30/174): rbd-mirror-19.0.0-2884.g218a6914.el9. 8.2 MB/s | 3.1 MB 00:00 2024-04-13T03:19:43.259 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 32/198 2024-04-13T03:19:43.287 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-5.4.4-4.el9.x86_64 33/198 2024-04-13T03:19:43.356 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 34/198 2024-04-13T03:19:43.409 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 35/198 2024-04-13T03:19:43.440 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 36/198 2024-04-13T03:19:43.475 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 37/198 2024-04-13T03:19:43.552 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 38/198 2024-04-13T03:19:43.553 INFO:teuthology.orchestra.run.smithi119.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2884.g218a6 7.9 MB/s | 3.5 MB 00:00 2024-04-13T03:19:43.595 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:19.0.0-2884.g218a6914.el9. 39/198 2024-04-13T03:19:43.629 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:19.0.0-2884.g218a6914.el 40/198 2024-04-13T03:19:43.645 INFO:teuthology.orchestra.run.smithi119.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2884.g21 2.7 MB/s | 249 kB 00:00 2024-04-13T03:19:43.666 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:19.0.0-2884.g218a6914.el9.x86_6 41/198 2024-04-13T03:19:43.712 INFO:teuthology.orchestra.run.smithi119.stdout:(33/174): ceph-mgr-rook-19.0.0-2884.g218a6914.e 738 kB/s | 49 kB 00:00 2024-04-13T03:19:43.735 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-13T03:19:43.770 INFO:teuthology.orchestra.run.smithi016.stdout:(1/174): ceph-19.0.0-2884.g218a6914.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-13T03:19:43.779 INFO:teuthology.orchestra.run.smithi119.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2884.g2 246 kB/s | 16 kB 00:00 2024-04-13T03:19:43.781 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-13T03:19:43.854 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-13T03:19:43.880 INFO:teuthology.orchestra.run.smithi119.stdout:(35/174): ceph-volume-19.0.0-2884.g218a6914.el9 2.6 MB/s | 264 kB 00:00 2024-04-13T03:19:43.987 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-13T03:19:44.041 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-13T03:19:44.063 INFO:teuthology.orchestra.run.smithi119.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 8.9 MB/s | 7.4 MB 00:00 2024-04-13T03:19:44.097 INFO:teuthology.orchestra.run.smithi119.stdout:(37/174): cephadm-19.0.0-2884.g218a6914.el9.noa 3.4 MB/s | 763 kB 00:00 2024-04-13T03:19:44.147 INFO:teuthology.orchestra.run.smithi119.stdout:(38/174): cephfs-top-19.0.0-2884.g218a6914.el9. 218 kB/s | 18 kB 00:00 2024-04-13T03:19:44.181 INFO:teuthology.orchestra.run.smithi119.stdout:(39/174): libaio-devel-0.3.112-1.el8.x86_64.rpm 210 kB/s | 18 kB 00:00 2024-04-13T03:19:44.264 INFO:teuthology.orchestra.run.smithi119.stdout:(40/174): fuse-2.9.9-15.el9.x86_64.rpm 694 kB/s | 80 kB 00:00 2024-04-13T03:19:44.348 INFO:teuthology.orchestra.run.smithi119.stdout:(41/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-13T03:19:44.432 INFO:teuthology.orchestra.run.smithi119.stdout:(42/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-13T03:19:44.524 INFO:teuthology.orchestra.run.smithi119.stdout:(43/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-04-13T03:19:44.537 INFO:teuthology.orchestra.run.smithi016.stdout:(2/174): ceph-base-19.0.0-2884.g218a6914.el9.x8 4.5 MB/s | 5.4 MB 00:01 2024-04-13T03:19:44.564 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-13T03:19:44.579 INFO:teuthology.orchestra.run.smithi016.stdout:(3/174): ceph-fuse-19.0.0-2884.g218a6914.el9.x8 1.1 MB/s | 891 kB 00:00 2024-04-13T03:19:44.607 INFO:teuthology.orchestra.run.smithi119.stdout:(44/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-13T03:19:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:(4/174): ceph-immutable-object-cache-19.0.0-288 1.9 MB/s | 145 kB 00:00 2024-04-13T03:19:44.645 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-13T03:19:44.674 INFO:teuthology.orchestra.run.smithi119.stdout:(45/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 899 kB/s | 218 kB 00:00 2024-04-13T03:19:44.721 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-13T03:19:44.758 INFO:teuthology.orchestra.run.smithi119.stdout:(46/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-04-13T03:19:44.774 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-13T03:19:44.784 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-13T03:19:44.814 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-13T03:19:44.829 INFO:teuthology.orchestra.run.smithi016.stdout:(5/174): ceph-mgr-19.0.0-2884.g218a6914.el9.x86 6.9 MB/s | 1.5 MB 00:00 2024-04-13T03:19:44.842 INFO:teuthology.orchestra.run.smithi119.stdout:(47/174): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-13T03:19:44.854 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-13T03:19:44.892 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:19.0.0-2884.g218a6914.el9.x86_6 53/198 2024-04-13T03:19:44.922 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-13T03:19:44.925 INFO:teuthology.orchestra.run.smithi119.stdout:(48/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-13T03:19:45.034 INFO:teuthology.orchestra.run.smithi119.stdout:(49/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-04-13T03:19:45.138 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-13T03:19:45.271 INFO:teuthology.orchestra.run.smithi016.stdout:(6/174): ceph-mon-19.0.0-2884.g218a6914.el9.x86 11 MB/s | 4.7 MB 00:00 2024-04-13T03:19:45.323 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-13T03:19:45.375 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-13T03:19:45.411 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-13T03:19:45.485 INFO:teuthology.orchestra.run.smithi119.stdout:(50/174): ceph-test-19.0.0-2884.g218a6914.el9.x 12 MB/s | 49 MB 00:04 2024-04-13T03:19:45.518 INFO:teuthology.orchestra.run.smithi119.stdout:(51/174): python3-cryptography-36.0.1-4.el9.x86 1.3 MB/s | 1.2 MB 00:00 2024-04-13T03:19:45.546 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-13T03:19:45.632 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-13T03:19:45.635 INFO:teuthology.orchestra.run.smithi119.stdout:(52/174): automake-1.16.2-8.el9.noarch.rpm 1.1 MB/s | 689 kB 00:00 2024-04-13T03:19:45.660 INFO:teuthology.orchestra.run.smithi119.stdout:(53/174): boost-program-options-1.75.0-8.el9.x8 753 kB/s | 107 kB 00:00 2024-04-13T03:19:45.663 INFO:teuthology.orchestra.run.smithi016.stdout:(7/174): ceph-mds-19.0.0-2884.g218a6914.el9.x86 2.2 MB/s | 2.4 MB 00:01 2024-04-13T03:19:45.685 INFO:teuthology.orchestra.run.smithi119.stdout:(54/174): elfutils-libelf-devel-0.190-2.el9.x86 428 kB/s | 21 kB 00:00 2024-04-13T03:19:45.727 INFO:teuthology.orchestra.run.smithi119.stdout:(55/174): flexiblas-3.0.4-8.el9.x86_64.rpm 790 kB/s | 33 kB 00:00 2024-04-13T03:19:45.753 INFO:teuthology.orchestra.run.smithi119.stdout:(56/174): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 943 kB 00:00 2024-04-13T03:19:45.778 INFO:teuthology.orchestra.run.smithi119.stdout:(57/174): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-13T03:19:45.803 INFO:teuthology.orchestra.run.smithi119.stdout:(58/174): flexiblas-openblas-openmp-3.0.4-8.el9 362 kB/s | 18 kB 00:00 2024-04-13T03:19:45.829 INFO:teuthology.orchestra.run.smithi119.stdout:(59/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-13T03:19:45.840 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-13T03:19:45.854 INFO:teuthology.orchestra.run.smithi119.stdout:(60/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 900 kB/s | 45 kB 00:00 2024-04-13T03:19:45.904 INFO:teuthology.orchestra.run.smithi119.stdout:(61/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-04-13T03:19:45.938 INFO:teuthology.orchestra.run.smithi119.stdout:(62/174): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 582 kB 00:00 2024-04-13T03:19:45.963 INFO:teuthology.orchestra.run.smithi119.stdout:(63/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-04-13T03:19:45.988 INFO:teuthology.orchestra.run.smithi119.stdout:(64/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-13T03:19:46.014 INFO:teuthology.orchestra.run.smithi119.stdout:(65/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-13T03:19:46.039 INFO:teuthology.orchestra.run.smithi119.stdout:(66/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-13T03:19:46.064 INFO:teuthology.orchestra.run.smithi119.stdout:(67/174): openblas-0.3.21-2.el9.x86_64.rpm 701 kB/s | 35 kB 00:00 2024-04-13T03:19:46.076 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-13T03:19:46.123 INFO:teuthology.orchestra.run.smithi119.stdout:(68/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.6 MB/s | 3.0 MB 00:00 2024-04-13T03:19:46.215 INFO:teuthology.orchestra.run.smithi119.stdout:(69/174): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-13T03:19:46.248 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-13T03:19:46.257 INFO:teuthology.orchestra.run.smithi119.stdout:(70/174): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-13T03:19:46.285 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-13T03:19:46.298 INFO:teuthology.orchestra.run.smithi119.stdout:(71/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-13T03:19:46.315 INFO:teuthology.orchestra.run.smithi119.stdout:(72/174): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-13T03:19:46.331 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-13T03:19:46.371 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-13T03:19:46.374 INFO:teuthology.orchestra.run.smithi119.stdout:(73/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 13 MB/s | 4.1 MB 00:00 2024-04-13T03:19:46.408 INFO:teuthology.orchestra.run.smithi119.stdout:(74/174): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-13T03:19:46.414 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-13T03:19:46.433 INFO:teuthology.orchestra.run.smithi119.stdout:(75/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-13T03:19:46.456 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-13T03:19:46.476 INFO:teuthology.orchestra.run.smithi119.stdout:(76/174): python3-devel-3.9.17-2.el9.x86_64.rpm 5.8 MB/s | 245 kB 00:00 2024-04-13T03:19:46.505 INFO:teuthology.orchestra.run.smithi016.stdout:(8/174): ceph-osd-19.0.0-2884.g218a6914.el9.x86 14 MB/s | 17 MB 00:01 2024-04-13T03:19:46.550 INFO:teuthology.orchestra.run.smithi119.stdout:(77/174): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-04-13T03:19:46.564 INFO:teuthology.orchestra.run.smithi016.stdout:(9/174): ceph-selinux-19.0.0-2884.g218a6914.el9 438 kB/s | 25 kB 00:00 2024-04-13T03:19:46.576 INFO:teuthology.orchestra.run.smithi119.stdout:(78/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-13T03:19:46.592 INFO:teuthology.orchestra.run.smithi119.stdout:(79/174): python3-jsonpatch-1.21-16.el9.noarch. 615 kB/s | 26 kB 00:00 2024-04-13T03:19:46.599 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-13T03:19:46.610 INFO:teuthology.orchestra.run.smithi119.stdout:(80/174): python3-jsonpointer-2.0-4.el9.noarch. 569 kB/s | 19 kB 00:00 2024-04-13T03:19:46.635 INFO:teuthology.orchestra.run.smithi119.stdout:(81/174): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-04-13T03:19:46.645 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-13T03:19:46.652 INFO:teuthology.orchestra.run.smithi119.stdout:(82/174): python3-markupsafe-1.1.1-12.el9.x86_6 832 kB/s | 35 kB 00:00 2024-04-13T03:19:46.678 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-13T03:19:46.717 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-13T03:19:46.735 INFO:teuthology.orchestra.run.smithi119.stdout:(83/174): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-04-13T03:19:46.761 INFO:teuthology.orchestra.run.smithi119.stdout:(84/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-13T03:19:46.786 INFO:teuthology.orchestra.run.smithi119.stdout:(85/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-13T03:19:46.811 INFO:teuthology.orchestra.run.smithi119.stdout:(86/174): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-13T03:19:46.812 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-13T03:19:46.828 INFO:teuthology.orchestra.run.smithi119.stdout:(87/174): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-13T03:19:46.851 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-13T03:19:46.853 INFO:teuthology.orchestra.run.smithi119.stdout:(88/174): python3-protobuf-3.14.0-13.el9.noarch 6.3 MB/s | 269 kB 00:00 2024-04-13T03:19:46.882 INFO:teuthology.orchestra.run.smithi119.stdout:(89/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-13T03:19:46.882 INFO:teuthology.orchestra.run.smithi016.stdout:(10/174): ceph-common-19.0.0-2884.g218a6914.el9 5.9 MB/s | 21 MB 00:03 2024-04-13T03:19:46.891 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-13T03:19:46.912 INFO:teuthology.orchestra.run.smithi119.stdout:(90/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-13T03:19:46.916 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-13T03:19:46.931 INFO:teuthology.orchestra.run.smithi016.stdout:(11/174): cephfs-mirror-19.0.0-2884.g218a6914.e 4.5 MB/s | 225 kB 00:00 2024-04-13T03:19:46.937 INFO:teuthology.orchestra.run.smithi119.stdout:(91/174): python3-pytz-2021.1-5.el9.noarch.rpm 870 kB/s | 51 kB 00:00 2024-04-13T03:19:46.942 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-13T03:19:46.963 INFO:teuthology.orchestra.run.smithi119.stdout:(92/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-13T03:19:47.009 INFO:teuthology.orchestra.run.smithi016.stdout:(12/174): cephfs-shell-19.0.0-2884.g218a6914.el 352 kB/s | 27 kB 00:00 2024-04-13T03:19:47.013 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-13T03:19:47.050 INFO:teuthology.orchestra.run.smithi119.stdout:(93/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-13T03:19:47.065 INFO:teuthology.orchestra.run.smithi016.stdout:(13/174): libcephfs-devel-19.0.0-2884.g218a6914 584 kB/s | 32 kB 00:00 2024-04-13T03:19:47.067 INFO:teuthology.orchestra.run.smithi119.stdout:(94/174): python3-toml-0.10.2-6.el9.noarch.rpm 400 kB/s | 42 kB 00:00 2024-04-13T03:19:47.086 INFO:teuthology.orchestra.run.smithi119.stdout:(95/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.3 MB/s | 44 kB 00:00 2024-04-13T03:19:47.109 INFO:teuthology.orchestra.run.smithi119.stdout:(96/174): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-04-13T03:19:47.126 INFO:teuthology.orchestra.run.smithi119.stdout:(97/174): qatlib-service-24.02.0-1.el9.x86_64.r 861 kB/s | 36 kB 00:00 2024-04-13T03:19:47.133 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-13T03:19:47.143 INFO:teuthology.orchestra.run.smithi119.stdout:(98/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-13T03:19:47.166 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-13T03:19:47.168 INFO:teuthology.orchestra.run.smithi119.stdout:(99/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 305 kB 00:00 2024-04-13T03:19:47.185 INFO:teuthology.orchestra.run.smithi119.stdout:(100/174): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-04-13T03:19:47.193 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-04-13T03:19:47.215 INFO:teuthology.orchestra.run.smithi016.stdout:(14/174): libcephfs2-19.0.0-2884.g218a6914.el9. 4.9 MB/s | 746 kB 00:00 2024-04-13T03:19:47.227 INFO:teuthology.orchestra.run.smithi119.stdout:(101/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-04-13T03:19:47.228 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 82/198 2024-04-13T03:19:47.244 INFO:teuthology.orchestra.run.smithi119.stdout:(102/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-13T03:19:47.261 INFO:teuthology.orchestra.run.smithi119.stdout:(103/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-13T03:19:47.267 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : uuid-1.6.2-55.el9.x86_64 83/198 2024-04-13T03:19:47.299 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 84/198 2024-04-13T03:19:47.307 INFO:teuthology.orchestra.run.smithi016.stdout:(15/174): libcephsqlite-19.0.0-2884.g218a6914.e 1.8 MB/s | 163 kB 00:00 2024-04-13T03:19:47.328 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 85/198 2024-04-13T03:19:47.345 INFO:teuthology.orchestra.run.smithi119.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-04-13T03:19:47.363 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 86/198 2024-04-13T03:19:47.398 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 87/198 2024-04-13T03:19:47.438 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 88/198 2024-04-13T03:19:47.473 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 89/198 2024-04-13T03:19:47.499 INFO:teuthology.orchestra.run.smithi016.stdout:(16/174): ceph-radosgw-19.0.0-2884.g218a6914.el 5.4 MB/s | 10 MB 00:01 2024-04-13T03:19:47.520 INFO:teuthology.orchestra.run.smithi119.stdout:(105/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-04-13T03:19:47.524 INFO:teuthology.orchestra.run.smithi016.stdout:(17/174): librados-devel-19.0.0-2884.g218a6914. 586 kB/s | 126 kB 00:00 2024-04-13T03:19:47.560 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 90/198 2024-04-13T03:19:47.574 INFO:teuthology.orchestra.run.smithi016.stdout:(18/174): libradosstriper1-19.0.0-2884.g218a691 6.6 MB/s | 500 kB 00:00 2024-04-13T03:19:47.603 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 91/198 2024-04-13T03:19:47.620 INFO:teuthology.orchestra.run.smithi119.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-04-13T03:19:47.648 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 92/198 2024-04-13T03:19:47.698 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 93/198 2024-04-13T03:19:47.699 INFO:teuthology.orchestra.run.smithi016.stdout:(19/174): python3-ceph-argparse-19.0.0-2884.g21 362 kB/s | 45 kB 00:00 2024-04-13T03:19:47.795 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-packaging-20.9-5.el9.noarch 94/198 2024-04-13T03:19:47.817 INFO:teuthology.orchestra.run.smithi016.stdout:(20/174): python3-ceph-common-19.0.0-2884.g218a 1.1 MB/s | 135 kB 00:00 2024-04-13T03:19:47.844 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 95/198 2024-04-13T03:19:47.887 INFO:teuthology.orchestra.run.smithi119.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 81 kB/s | 19 kB 00:00 2024-04-13T03:19:47.942 INFO:teuthology.orchestra.run.smithi016.stdout:(21/174): python3-cephfs-19.0.0-2884.g218a6914. 1.3 MB/s | 162 kB 00:00 2024-04-13T03:19:47.946 INFO:teuthology.orchestra.run.smithi119.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 913 kB/s | 548 kB 00:00 2024-04-13T03:19:47.980 INFO:teuthology.orchestra.run.smithi119.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 715 kB/s | 309 kB 00:00 2024-04-13T03:19:48.005 INFO:teuthology.orchestra.run.smithi119.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-04-13T03:19:48.047 INFO:teuthology.orchestra.run.smithi119.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 727 kB/s | 49 kB 00:00 2024-04-13T03:19:48.075 INFO:teuthology.orchestra.run.smithi016.stdout:(22/174): librgw2-19.0.0-2884.g218a6914.el9.x86 9.5 MB/s | 5.2 MB 00:00 2024-04-13T03:19:48.080 INFO:teuthology.orchestra.run.smithi119.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-04-13T03:19:48.117 INFO:teuthology.orchestra.run.smithi016.stdout:(23/174): python3-rados-19.0.0-2884.g218a6914.e 1.8 MB/s | 321 kB 00:00 2024-04-13T03:19:48.131 INFO:teuthology.orchestra.run.smithi119.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-13T03:19:48.159 INFO:teuthology.orchestra.run.smithi016.stdout:(24/174): python3-rbd-19.0.0-2884.g218a6914.el9 3.5 MB/s | 298 kB 00:00 2024-04-13T03:19:48.185 INFO:teuthology.orchestra.run.smithi016.stdout:(25/174): python3-rgw-19.0.0-2884.g218a6914.el9 1.5 MB/s | 100 kB 00:00 2024-04-13T03:19:48.210 INFO:teuthology.orchestra.run.smithi016.stdout:(26/174): rbd-fuse-19.0.0-2884.g218a6914.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-13T03:19:48.214 INFO:teuthology.orchestra.run.smithi119.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-04-13T03:19:48.269 INFO:teuthology.orchestra.run.smithi016.stdout:(27/174): rbd-nbd-19.0.0-2884.g218a6914.el9.x86 2.9 MB/s | 170 kB 00:00 2024-04-13T03:19:48.289 INFO:teuthology.orchestra.run.smithi119.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-04-13T03:19:48.306 INFO:teuthology.orchestra.run.smithi119.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-04-13T03:19:48.319 INFO:teuthology.orchestra.run.smithi016.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2884.g 700 kB/s | 34 kB 00:00 2024-04-13T03:19:48.348 INFO:teuthology.orchestra.run.smithi119.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-04-13T03:19:48.394 INFO:teuthology.orchestra.run.smithi016.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2884.g218a691 1.9 MB/s | 142 kB 00:00 2024-04-13T03:19:48.415 INFO:teuthology.orchestra.run.smithi119.stdout:(118/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-04-13T03:19:48.432 INFO:teuthology.orchestra.run.smithi119.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 119 kB/s | 15 kB 00:00 2024-04-13T03:19:48.457 INFO:teuthology.orchestra.run.smithi119.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-13T03:19:48.499 INFO:teuthology.orchestra.run.smithi119.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-13T03:19:48.503 INFO:teuthology.orchestra.run.smithi016.stdout:(30/174): rbd-mirror-19.0.0-2884.g218a6914.el9. 9.8 MB/s | 3.1 MB 00:00 2024-04-13T03:19:48.525 INFO:teuthology.orchestra.run.smithi119.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-13T03:19:48.550 INFO:teuthology.orchestra.run.smithi119.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-04-13T03:19:48.575 INFO:teuthology.orchestra.run.smithi119.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-04-13T03:19:48.576 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 96/198 2024-04-13T03:19:48.625 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 97/198 2024-04-13T03:19:48.650 INFO:teuthology.orchestra.run.smithi119.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-04-13T03:19:48.659 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 98/198 2024-04-13T03:19:48.676 INFO:teuthology.orchestra.run.smithi119.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-13T03:19:48.701 INFO:teuthology.orchestra.run.smithi119.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 960 kB/s | 144 kB 00:00 2024-04-13T03:19:48.726 INFO:teuthology.orchestra.run.smithi119.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-04-13T03:19:48.751 INFO:teuthology.orchestra.run.smithi119.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-04-13T03:19:48.757 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 99/198 2024-04-13T03:19:48.777 INFO:teuthology.orchestra.run.smithi119.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-13T03:19:48.802 INFO:teuthology.orchestra.run.smithi119.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-13T03:19:48.808 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 100/198 2024-04-13T03:19:48.837 INFO:teuthology.orchestra.run.smithi016.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2884.g218a6 8.0 MB/s | 3.5 MB 00:00 2024-04-13T03:19:48.844 INFO:teuthology.orchestra.run.smithi119.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-13T03:19:48.869 INFO:teuthology.orchestra.run.smithi119.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-04-13T03:19:48.894 INFO:teuthology.orchestra.run.smithi119.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 858 kB/s | 79 kB 00:00 2024-04-13T03:19:48.895 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 101/198 2024-04-13T03:19:48.912 INFO:teuthology.orchestra.run.smithi016.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2884.g21 3.3 MB/s | 249 kB 00:00 2024-04-13T03:19:48.919 INFO:teuthology.orchestra.run.smithi119.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 764 kB/s | 58 kB 00:00 2024-04-13T03:19:48.936 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 102/198 2024-04-13T03:19:48.953 INFO:teuthology.orchestra.run.smithi119.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-13T03:19:48.978 INFO:teuthology.orchestra.run.smithi119.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-04-13T03:19:48.987 INFO:teuthology.orchestra.run.smithi016.stdout:(33/174): ceph-mgr-rook-19.0.0-2884.g218a6914.e 663 kB/s | 49 kB 00:00 2024-04-13T03:19:49.004 INFO:teuthology.orchestra.run.smithi119.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-04-13T03:19:49.021 INFO:teuthology.orchestra.run.smithi119.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 420 kB/s | 28 kB 00:00 2024-04-13T03:19:49.046 INFO:teuthology.orchestra.run.smithi119.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 458 kB/s | 31 kB 00:00 2024-04-13T03:19:49.062 INFO:teuthology.orchestra.run.smithi016.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2884.g2 222 kB/s | 16 kB 00:00 2024-04-13T03:19:49.088 INFO:teuthology.orchestra.run.smithi119.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-13T03:19:49.105 INFO:teuthology.orchestra.run.smithi119.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 703 kB/s | 59 kB 00:00 2024-04-13T03:19:49.115 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.21-2.el9.x86_64 103/198 2024-04-13T03:19:49.122 INFO:teuthology.orchestra.run.smithi119.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 472 kB/s | 36 kB 00:00 2024-04-13T03:19:49.142 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 104/198 2024-04-13T03:19:49.154 INFO:teuthology.orchestra.run.smithi016.stdout:(35/174): ceph-volume-19.0.0-2884.g218a6914.el9 2.8 MB/s | 264 kB 00:00 2024-04-13T03:19:49.164 INFO:teuthology.orchestra.run.smithi119.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 679 kB/s | 51 kB 00:00 2024-04-13T03:19:49.189 INFO:teuthology.orchestra.run.smithi119.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-13T03:19:49.214 INFO:teuthology.orchestra.run.smithi119.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 973 kB/s | 90 kB 00:00 2024-04-13T03:19:49.227 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 105/198 2024-04-13T03:19:49.239 INFO:teuthology.orchestra.run.smithi119.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-13T03:19:49.256 INFO:teuthology.orchestra.run.smithi119.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-04-13T03:19:49.290 INFO:teuthology.orchestra.run.smithi119.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-13T03:19:49.313 INFO:teuthology.orchestra.run.smithi016.stdout:(36/174): cephadm-19.0.0-2884.g218a6914.el9.noa 4.7 MB/s | 763 kB 00:00 2024-04-13T03:19:49.370 INFO:teuthology.orchestra.run.smithi119.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-13T03:19:49.388 INFO:teuthology.orchestra.run.smithi016.stdout:(37/174): cephfs-top-19.0.0-2884.g218a6914.el9. 246 kB/s | 18 kB 00:00 2024-04-13T03:19:49.580 INFO:teuthology.orchestra.run.smithi016.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 6.9 MB/s | 7.4 MB 00:01 2024-04-13T03:19:49.623 INFO:teuthology.orchestra.run.smithi016.stdout:(39/174): libaio-devel-0.3.112-1.el8.x86_64.rpm 75 kB/s | 18 kB 00:00 2024-04-13T03:19:49.791 INFO:teuthology.orchestra.run.smithi119.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 5.9 MB/s | 2.3 MB 00:00 2024-04-13T03:19:49.849 INFO:teuthology.orchestra.run.smithi119.stdout:(152/174): librados2-19.0.0-2884.g218a6914.el9. 5.7 MB/s | 3.4 MB 00:00 2024-04-13T03:19:49.855 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 106/198 2024-04-13T03:19:49.899 INFO:teuthology.orchestra.run.smithi119.stdout:(153/174): librbd1-19.0.0-2884.g218a6914.el9.x8 5.2 MB/s | 3.1 MB 00:00 2024-04-13T03:19:49.925 INFO:teuthology.orchestra.run.smithi119.stdout:(154/174): NetworkManager-team-1.47.4-1.el9.x86 556 kB/s | 42 kB 00:00 2024-04-13T03:19:49.973 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 107/198 2024-04-13T03:19:49.975 INFO:teuthology.orchestra.run.smithi119.stdout:(155/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-04-13T03:19:50.033 INFO:teuthology.orchestra.run.smithi119.stdout:(156/174): elfutils-debuginfod-client-0.190-2.e 627 kB/s | 36 kB 00:00 2024-04-13T03:19:50.075 INFO:teuthology.orchestra.run.smithi119.stdout:(157/174): NetworkManager-libnm-1.47.4-1.el9.x8 6.4 MB/s | 1.8 MB 00:00 2024-04-13T03:19:50.101 INFO:teuthology.orchestra.run.smithi119.stdout:(158/174): NetworkManager-tui-1.47.4-1.el9.x86_ 1.2 MB/s | 249 kB 00:00 2024-04-13T03:19:50.126 INFO:teuthology.orchestra.run.smithi119.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-04-13T03:19:50.151 INFO:teuthology.orchestra.run.smithi119.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.3 MB/s | 256 kB 00:00 2024-04-13T03:19:50.176 INFO:teuthology.orchestra.run.smithi119.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 305 kB/s | 23 kB 00:00 2024-04-13T03:19:50.202 INFO:teuthology.orchestra.run.smithi119.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-04-13T03:19:50.223 INFO:teuthology.orchestra.run.smithi016.stdout:(40/174): ceph-test-19.0.0-2884.g218a6914.el9.x 13 MB/s | 49 MB 00:03 2024-04-13T03:19:50.227 INFO:teuthology.orchestra.run.smithi119.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-04-13T03:19:50.252 INFO:teuthology.orchestra.run.smithi119.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-04-13T03:19:50.277 INFO:teuthology.orchestra.run.smithi119.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-04-13T03:19:50.302 INFO:teuthology.orchestra.run.smithi119.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-04-13T03:19:50.406 INFO:teuthology.orchestra.run.smithi016.stdout:(41/174): fuse-2.9.9-15.el9.x86_64.rpm 98 kB/s | 80 kB 00:00 2024-04-13T03:19:50.431 INFO:teuthology.orchestra.run.smithi016.stdout:(42/174): jq-1.6-16.el9.x86_64.rpm 232 kB/s | 187 kB 00:00 2024-04-13T03:19:50.495 INFO:teuthology.orchestra.run.smithi119.stdout:(167/174): util-linux-2.37.4-18.el9.x86_64.rpm 12 MB/s | 2.3 MB 00:00 2024-04-13T03:19:50.523 INFO:teuthology.orchestra.run.smithi016.stdout:(43/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 218 kB 00:00 2024-04-13T03:19:50.528 INFO:teuthology.orchestra.run.smithi119.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.4 MB/s | 1.2 MB 00:00 2024-04-13T03:19:50.548 INFO:teuthology.orchestra.run.smithi016.stdout:(44/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-04-13T03:19:50.553 INFO:teuthology.orchestra.run.smithi119.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 7.8 MB/s | 465 kB 00:00 2024-04-13T03:19:50.579 INFO:teuthology.orchestra.run.smithi119.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 620 kB/s | 31 kB 00:00 2024-04-13T03:19:50.590 INFO:teuthology.orchestra.run.smithi016.stdout:(45/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-04-13T03:19:50.612 INFO:teuthology.orchestra.run.smithi119.stdout:(171/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-04-13T03:19:50.624 INFO:teuthology.orchestra.run.smithi016.stdout:(46/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-04-13T03:19:50.657 INFO:teuthology.orchestra.run.smithi016.stdout:(47/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-13T03:19:50.691 INFO:teuthology.orchestra.run.smithi016.stdout:(48/174): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-13T03:19:50.696 INFO:teuthology.orchestra.run.smithi119.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 889 kB/s | 74 kB 00:00 2024-04-13T03:19:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:(49/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-04-13T03:19:50.754 INFO:teuthology.orchestra.run.smithi119.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.1 MB/s | 1.1 MB 00:00 2024-04-13T03:19:50.783 INFO:teuthology.orchestra.run.smithi016.stdout:(50/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-04-13T03:19:50.842 INFO:teuthology.orchestra.run.smithi016.stdout:(51/174): python3-cryptography-36.0.1-4.el9.x86 4.2 MB/s | 1.2 MB 00:00 2024-04-13T03:19:50.913 INFO:teuthology.orchestra.run.smithi119.stdout:(174/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.4 MB/s | 2.2 MB 00:00 2024-04-13T03:19:50.919 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-13T03:19:50.919 INFO:teuthology.orchestra.run.smithi119.stdout:Total 18 MB/s | 229 MB 00:12 2024-04-13T03:19:50.978 INFO:teuthology.orchestra.run.smithi016.stdout:(52/174): automake-1.16.2-8.el9.noarch.rpm 3.0 MB/s | 689 kB 00:00 2024-04-13T03:19:51.001 INFO:teuthology.orchestra.run.smithi016.stdout:(53/174): boost-program-options-1.75.0-8.el9.x8 678 kB/s | 107 kB 00:00 2024-04-13T03:19:51.018 INFO:teuthology.orchestra.run.smithi016.stdout:(54/174): elfutils-libelf-devel-0.190-2.el9.x86 522 kB/s | 21 kB 00:00 2024-04-13T03:19:51.102 INFO:teuthology.orchestra.run.smithi016.stdout:(55/174): bison-3.7.4-5.el9.x86_64.rpm 2.9 MB/s | 943 kB 00:00 2024-04-13T03:19:51.119 INFO:teuthology.orchestra.run.smithi016.stdout:(56/174): flexiblas-3.0.4-8.el9.x86_64.rpm 328 kB/s | 33 kB 00:00 2024-04-13T03:19:51.152 INFO:teuthology.orchestra.run.smithi016.stdout:(57/174): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 313 kB 00:00 2024-04-13T03:19:51.169 INFO:teuthology.orchestra.run.smithi016.stdout:(58/174): flexiblas-openblas-openmp-3.0.4-8.el9 368 kB/s | 18 kB 00:00 2024-04-13T03:19:51.188 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 108/198 2024-04-13T03:19:51.203 INFO:teuthology.orchestra.run.smithi016.stdout:(59/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 160 kB 00:00 2024-04-13T03:19:51.221 INFO:teuthology.orchestra.run.smithi016.stdout:(60/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 915 kB/s | 45 kB 00:00 2024-04-13T03:19:51.280 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 109/198 2024-04-13T03:19:51.295 INFO:teuthology.orchestra.run.smithi016.stdout:(61/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 662 kB 00:00 2024-04-13T03:19:51.306 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 110/198 2024-04-13T03:19:51.329 INFO:teuthology.orchestra.run.smithi016.stdout:(62/174): libtool-2.4.6-45.el9.x86_64.rpm 5.3 MB/s | 582 kB 00:00 2024-04-13T03:19:51.345 INFO:teuthology.orchestra.run.smithi016.stdout:(63/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 517 kB/s | 25 kB 00:00 2024-04-13T03:19:51.371 INFO:teuthology.orchestra.run.smithi016.stdout:(64/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-13T03:19:51.429 INFO:teuthology.orchestra.run.smithi016.stdout:(65/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-13T03:19:51.505 INFO:teuthology.orchestra.run.smithi016.stdout:(66/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.4 MB/s | 3.0 MB 00:00 2024-04-13T03:19:51.521 INFO:teuthology.orchestra.run.smithi016.stdout:(67/174): openblas-0.3.21-2.el9.x86_64.rpm 387 kB/s | 35 kB 00:00 2024-04-13T03:19:51.547 INFO:teuthology.orchestra.run.smithi016.stdout:(68/174): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-04-13T03:19:51.619 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 111/198 2024-04-13T03:19:51.653 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 112/198 2024-04-13T03:19:51.672 INFO:teuthology.orchestra.run.smithi016.stdout:(69/174): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-13T03:19:51.743 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 112/198 2024-04-13T03:19:51.775 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:19.0.0-2884.g218a6914.el9.x86_64 113/198 2024-04-13T03:19:51.789 INFO:teuthology.orchestra.run.smithi016.stdout:(70/174): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-04-13T03:19:51.811 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 114/198 2024-04-13T03:19:51.881 INFO:teuthology.orchestra.run.smithi016.stdout:(71/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 12 MB/s | 4.1 MB 00:00 2024-04-13T03:19:51.931 INFO:teuthology.orchestra.run.smithi016.stdout:(72/174): postgresql-contrib-13.14-1.el9.x86_64 3.4 MB/s | 911 kB 00:00 2024-04-13T03:19:51.965 INFO:teuthology.orchestra.run.smithi016.stdout:(73/174): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-13T03:19:52.032 INFO:teuthology.orchestra.run.smithi016.stdout:(74/174): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-13T03:19:52.077 INFO:teuthology.orchestra.run.smithi016.stdout:(75/174): python3-attrs-20.3.0-7.el9.noarch.rpm 787 kB/s | 87 kB 00:00 2024-04-13T03:19:52.149 INFO:teuthology.orchestra.run.smithi016.stdout:(76/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.4 MB/s | 245 kB 00:00 2024-04-13T03:19:52.215 INFO:teuthology.orchestra.run.smithi016.stdout:(77/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.7 MB/s | 248 kB 00:00 2024-04-13T03:19:52.254 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-13T03:19:52.266 INFO:teuthology.orchestra.run.smithi016.stdout:(78/174): python3-jsonpatch-1.21-16.el9.noarch. 521 kB/s | 26 kB 00:00 2024-04-13T03:19:52.280 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-13T03:19:52.306 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 116/198 2024-04-13T03:19:52.377 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 116/198 2024-04-13T03:19:52.410 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:19.0.0-2884.g218a6914.el9.x86_64 117/198 2024-04-13T03:19:52.431 INFO:teuthology.orchestra.run.smithi016.stdout:(79/174): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-04-13T03:19:52.458 INFO:teuthology.orchestra.run.smithi016.stdout:(80/174): python3-jsonpointer-2.0-4.el9.noarch. 100 kB/s | 19 kB 00:00 2024-04-13T03:19:52.492 INFO:teuthology.orchestra.run.smithi016.stdout:(81/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 172 kB 00:00 2024-04-13T03:19:52.517 INFO:teuthology.orchestra.run.smithi016.stdout:(82/174): python3-markupsafe-1.1.1-12.el9.x86_6 603 kB/s | 35 kB 00:00 2024-04-13T03:19:52.531 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-13T03:19:52.531 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-13T03:19:52.659 INFO:teuthology.orchestra.run.smithi016.stdout:(83/174): python3-babel-2.9.1-2.el9.noarch.rpm 9.5 MB/s | 6.0 MB 00:00 2024-04-13T03:19:52.693 INFO:teuthology.orchestra.run.smithi016.stdout:(84/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 250 kB 00:00 2024-04-13T03:19:52.768 INFO:teuthology.orchestra.run.smithi016.stdout:(85/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-04-13T03:19:52.793 INFO:teuthology.orchestra.run.smithi016.stdout:(86/174): python3-packaging-20.9-5.el9.noarch.r 772 kB/s | 77 kB 00:00 2024-04-13T03:19:52.810 INFO:teuthology.orchestra.run.smithi016.stdout:(87/174): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-13T03:19:52.853 INFO:teuthology.orchestra.run.smithi016.stdout:(88/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-04-13T03:19:52.886 INFO:teuthology.orchestra.run.smithi016.stdout:(89/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-13T03:19:52.920 INFO:teuthology.orchestra.run.smithi016.stdout:(90/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-13T03:19:52.937 INFO:teuthology.orchestra.run.smithi016.stdout:(91/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-13T03:19:52.962 INFO:teuthology.orchestra.run.smithi016.stdout:(92/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-04-13T03:19:53.012 INFO:teuthology.orchestra.run.smithi016.stdout:(93/174): python3-toml-0.10.2-6.el9.noarch.rpm 846 kB/s | 42 kB 00:00 2024-04-13T03:19:53.154 INFO:teuthology.orchestra.run.smithi016.stdout:(94/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.7 MB/s | 5.1 MB 00:00 2024-04-13T03:19:53.171 INFO:teuthology.orchestra.run.smithi016.stdout:(95/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 281 kB/s | 44 kB 00:00 2024-04-13T03:19:53.205 INFO:teuthology.orchestra.run.smithi016.stdout:(96/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.4 MB/s | 221 kB 00:00 2024-04-13T03:19:53.222 INFO:teuthology.orchestra.run.smithi016.stdout:(97/174): qatlib-service-24.02.0-1.el9.x86_64.r 726 kB/s | 36 kB 00:00 2024-04-13T03:19:53.247 INFO:teuthology.orchestra.run.smithi016.stdout:(98/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-13T03:19:53.289 INFO:teuthology.orchestra.run.smithi016.stdout:(99/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-13T03:19:53.306 INFO:teuthology.orchestra.run.smithi016.stdout:(100/174): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-04-13T03:19:53.340 INFO:teuthology.orchestra.run.smithi016.stdout:(101/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.9 MB/s | 94 kB 00:00 2024-04-13T03:19:53.365 INFO:teuthology.orchestra.run.smithi016.stdout:(102/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-13T03:19:53.390 INFO:teuthology.orchestra.run.smithi016.stdout:(103/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 918 kB/s | 45 kB 00:00 2024-04-13T03:19:53.599 INFO:teuthology.orchestra.run.smithi016.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 96 kB/s | 22 kB 00:00 2024-04-13T03:19:53.858 INFO:teuthology.orchestra.run.smithi016.stdout:(105/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-04-13T03:19:54.099 INFO:teuthology.orchestra.run.smithi016.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-04-13T03:19:54.334 INFO:teuthology.orchestra.run.smithi016.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 84 kB/s | 19 kB 00:00 2024-04-13T03:19:54.366 INFO:teuthology.orchestra.run.smithi016.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 609 kB/s | 309 kB 00:00 2024-04-13T03:19:54.409 INFO:teuthology.orchestra.run.smithi016.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 678 kB/s | 548 kB 00:00 2024-04-13T03:19:54.434 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-13T03:19:54.435 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-13T03:19:54.459 INFO:teuthology.orchestra.run.smithi016.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-04-13T03:19:54.534 INFO:teuthology.orchestra.run.smithi016.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 390 kB/s | 49 kB 00:00 2024-04-13T03:19:54.568 INFO:teuthology.orchestra.run.smithi016.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 623 kB/s | 67 kB 00:00 2024-04-13T03:19:54.622 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 118/198 2024-04-13T03:19:54.660 INFO:teuthology.orchestra.run.smithi016.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-13T03:19:54.663 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 118/198 2024-04-13T03:19:54.848 INFO:teuthology.orchestra.run.smithi016.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 838 kB 00:00 2024-04-13T03:19:54.885 INFO:teuthology.orchestra.run.smithi016.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.4 MB/s | 548 kB 00:00 2024-04-13T03:19:54.944 INFO:teuthology.orchestra.run.smithi016.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 457 kB/s | 43 kB 00:00 2024-04-13T03:19:54.978 INFO:teuthology.orchestra.run.smithi016.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 352 kB/s | 32 kB 00:00 2024-04-13T03:19:55.036 INFO:teuthology.orchestra.run.smithi016.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 164 kB/s | 15 kB 00:00 2024-04-13T03:19:55.146 INFO:teuthology.orchestra.run.smithi016.stdout:(119/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-04-13T03:19:55.202 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 118/198 2024-04-13T03:19:55.203 INFO:teuthology.orchestra.run.smithi016.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 764 kB/s | 172 kB 00:00 2024-04-13T03:19:55.237 INFO:teuthology.orchestra.run.smithi016.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.8 MB/s | 358 kB 00:00 2024-04-13T03:19:55.241 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 119/198 2024-04-13T03:19:55.271 INFO:teuthology.orchestra.run.smithi016.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-13T03:19:55.304 INFO:teuthology.orchestra.run.smithi016.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 354 kB/s | 35 kB 00:00 2024-04-13T03:19:55.322 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 119/198 2024-04-13T03:19:55.322 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-13T03:19:55.322 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-13T03:19:55.322 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:19:55.338 INFO:teuthology.orchestra.run.smithi016.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-04-13T03:19:55.351 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 120/198 2024-04-13T03:19:55.438 INFO:teuthology.orchestra.run.smithi016.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-04-13T03:19:55.464 INFO:teuthology.orchestra.run.smithi016.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 86 kB/s | 11 kB 00:00 2024-04-13T03:19:55.489 INFO:teuthology.orchestra.run.smithi016.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 787 kB/s | 144 kB 00:00 2024-04-13T03:19:55.547 INFO:teuthology.orchestra.run.smithi016.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-04-13T03:19:55.573 INFO:teuthology.orchestra.run.smithi016.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-04-13T03:19:55.598 INFO:teuthology.orchestra.run.smithi016.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-04-13T03:19:55.640 INFO:teuthology.orchestra.run.smithi016.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-04-13T03:19:55.698 INFO:teuthology.orchestra.run.smithi016.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.2 MB/s | 1.0 MB 00:00 2024-04-13T03:19:55.724 INFO:teuthology.orchestra.run.smithi016.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 370 kB/s | 46 kB 00:00 2024-04-13T03:19:55.749 INFO:teuthology.orchestra.run.smithi016.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 728 kB/s | 79 kB 00:00 2024-04-13T03:19:55.791 INFO:teuthology.orchestra.run.smithi016.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 631 kB/s | 58 kB 00:00 2024-04-13T03:19:55.874 INFO:teuthology.orchestra.run.smithi016.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-04-13T03:19:55.900 INFO:teuthology.orchestra.run.smithi016.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 109 kB/s | 16 kB 00:00 2024-04-13T03:19:55.925 INFO:teuthology.orchestra.run.smithi016.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 675 kB/s | 90 kB 00:00 2024-04-13T03:19:55.989 INFO:teuthology.orchestra.run.smithi016.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 249 kB/s | 28 kB 00:00 2024-04-13T03:19:56.014 INFO:teuthology.orchestra.run.smithi016.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 272 kB/s | 31 kB 00:00 2024-04-13T03:19:56.048 INFO:teuthology.orchestra.run.smithi016.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-13T03:19:56.089 INFO:teuthology.orchestra.run.smithi016.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 591 kB/s | 59 kB 00:00 2024-04-13T03:19:56.115 INFO:teuthology.orchestra.run.smithi016.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-04-13T03:19:56.140 INFO:teuthology.orchestra.run.smithi016.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-13T03:19:56.257 INFO:teuthology.orchestra.run.smithi016.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-04-13T03:19:56.274 INFO:teuthology.orchestra.run.smithi016.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 567 kB/s | 90 kB 00:00 2024-04-13T03:19:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-04-13T03:19:56.350 INFO:teuthology.orchestra.run.smithi016.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 219 kB/s | 20 kB 00:00 2024-04-13T03:19:56.383 INFO:teuthology.orchestra.run.smithi016.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-13T03:19:56.509 INFO:teuthology.orchestra.run.smithi016.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-04-13T03:19:56.934 INFO:teuthology.orchestra.run.smithi016.stdout:(151/174): librados2-19.0.0-2884.g218a6914.el9. 5.8 MB/s | 3.4 MB 00:00 2024-04-13T03:19:57.076 INFO:teuthology.orchestra.run.smithi016.stdout:(152/174): librbd1-19.0.0-2884.g218a6914.el9.x8 5.1 MB/s | 3.1 MB 00:00 2024-04-13T03:19:57.135 INFO:teuthology.orchestra.run.smithi016.stdout:(153/174): NetworkManager-1.47.4-1.el9.x86_64.r 4.1 MB/s | 2.3 MB 00:00 2024-04-13T03:19:57.151 INFO:teuthology.orchestra.run.smithi016.stdout:(154/174): NetworkManager-team-1.47.4-1.el9.x86 564 kB/s | 42 kB 00:00 2024-04-13T03:19:57.235 INFO:teuthology.orchestra.run.smithi016.stdout:(155/174): acl-2.3.1-4.el9.x86_64.rpm 863 kB/s | 71 kB 00:00 2024-04-13T03:19:57.285 INFO:teuthology.orchestra.run.smithi016.stdout:(156/174): elfutils-debuginfod-client-0.190-2.e 746 kB/s | 36 kB 00:00 2024-04-13T03:19:57.352 INFO:teuthology.orchestra.run.smithi016.stdout:(157/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.9 MB/s | 192 kB 00:00 2024-04-13T03:19:57.394 INFO:teuthology.orchestra.run.smithi016.stdout:(158/174): NetworkManager-tui-1.47.4-1.el9.x86_ 963 kB/s | 249 kB 00:00 2024-04-13T03:19:57.436 INFO:teuthology.orchestra.run.smithi016.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-04-13T03:19:57.461 INFO:teuthology.orchestra.run.smithi016.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 348 kB/s | 23 kB 00:00 2024-04-13T03:19:57.520 INFO:teuthology.orchestra.run.smithi016.stdout:(161/174): NetworkManager-libnm-1.47.4-1.el9.x8 3.1 MB/s | 1.8 MB 00:00 2024-04-13T03:19:57.562 INFO:teuthology.orchestra.run.smithi016.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 866 kB/s | 108 kB 00:00 2024-04-13T03:19:57.587 INFO:teuthology.orchestra.run.smithi016.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 155 kB 00:00 2024-04-13T03:19:57.612 INFO:teuthology.orchestra.run.smithi016.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 136 kB 00:00 2024-04-13T03:19:57.638 INFO:teuthology.orchestra.run.smithi016.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 857 kB/s | 64 kB 00:00 2024-04-13T03:19:57.663 INFO:teuthology.orchestra.run.smithi016.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 383 kB/s | 29 kB 00:00 2024-04-13T03:19:57.747 INFO:teuthology.orchestra.run.smithi016.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 9.1 MB/s | 1.2 MB 00:00 2024-04-13T03:19:57.814 INFO:teuthology.orchestra.run.smithi016.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-04-13T03:19:57.856 INFO:teuthology.orchestra.run.smithi016.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-04-13T03:19:57.906 INFO:teuthology.orchestra.run.smithi016.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 342 kB/s | 31 kB 00:00 2024-04-13T03:19:57.939 INFO:teuthology.orchestra.run.smithi016.stdout:(171/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2024-04-13T03:19:58.006 INFO:teuthology.orchestra.run.smithi016.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-04-13T03:19:58.031 INFO:teuthology.orchestra.run.smithi016.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 817 kB/s | 74 kB 00:00 2024-04-13T03:19:58.107 INFO:teuthology.orchestra.run.smithi016.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.2 MB/s | 2.3 MB 00:00 2024-04-13T03:19:58.116 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-13T03:19:58.116 INFO:teuthology.orchestra.run.smithi016.stdout:Total 15 MB/s | 229 MB 00:15 2024-04-13T03:20:01.479 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-13T03:20:01.592 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-13T03:20:01.626 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-13T03:20:01.666 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-13T03:20:01.675 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-13T03:20:01.723 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-13T03:20:01.776 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-13T03:20:01.810 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-13T03:20:01.836 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-13T03:20:01.874 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-13T03:20:01.878 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-13T03:20:02.199 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-13T03:20:02.253 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.0.0-2884.g218a6914.el9.x86_64 10/198 2024-04-13T03:20:02.288 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-13T03:20:02.288 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-13T03:20:02.357 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.0.0-2884.g218a6914.el9.x86_64 10/198 2024-04-13T03:20:02.396 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 11/198 2024-04-13T03:20:02.471 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 11/198 2024-04-13T03:20:02.500 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.0.0-2884.g218a6914.el9.x86_64 12/198 2024-04-13T03:20:02.539 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-13T03:20:02.717 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-13T03:20:02.764 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-13T03:20:02.792 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 16/198 2024-04-13T03:20:02.896 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 16/198 2024-04-13T03:20:02.927 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86 17/198 2024-04-13T03:20:02.990 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86 17/198 2024-04-13T03:20:03.022 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-13T03:20:03.045 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-13T03:20:03.063 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-13T03:20:03.245 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-13T03:20:03.276 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-13T03:20:03.308 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-13T03:20:03.414 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-13T03:20:03.518 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-13T03:20:03.578 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-13T03:20:03.639 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-13T03:20:03.696 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-13T03:20:03.910 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-13T03:20:04.018 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-13T03:20:04.112 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-13T03:20:04.143 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-13T03:20:04.177 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-13T03:20:04.266 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/198 2024-04-13T03:20:04.310 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/198 2024-04-13T03:20:04.353 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/198 2024-04-13T03:20:04.386 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 32/198 2024-04-13T03:20:04.414 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 33/198 2024-04-13T03:20:04.482 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 34/198 2024-04-13T03:20:04.536 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 35/198 2024-04-13T03:20:04.567 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 36/198 2024-04-13T03:20:04.602 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 37/198 2024-04-13T03:20:04.669 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 38/198 2024-04-13T03:20:04.705 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.0.0-2884.g218a6914.el9. 39/198 2024-04-13T03:20:04.749 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.0.0-2884.g218a6914.el 40/198 2024-04-13T03:20:04.777 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.0.0-2884.g218a6914.el9.x86_6 41/198 2024-04-13T03:20:04.846 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-13T03:20:04.864 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-13T03:20:04.865 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-13T03:20:04.891 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-13T03:20:04.966 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-13T03:20:05.100 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-13T03:20:05.152 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-13T03:20:05.682 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-13T03:20:05.772 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-13T03:20:05.860 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-13T03:20:05.909 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-13T03:20:05.919 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-13T03:20:05.949 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-13T03:20:05.998 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-13T03:20:06.036 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.0.0-2884.g218a6914.el9.x86_6 53/198 2024-04-13T03:20:06.065 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-13T03:20:06.297 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-13T03:20:06.464 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-13T03:20:06.572 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-13T03:20:06.746 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-13T03:20:06.922 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 120/198 2024-04-13T03:20:06.923 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-04-13T03:20:06.923 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-04-13T03:20:06.923 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-04-13T03:20:06.923 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-04-13T03:20:06.923 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-04-13T03:20:06.923 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /root 2024-04-13T03:20:06.923 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-04-13T03:20:06.923 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:07.081 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 121/198 2024-04-13T03:20:07.098 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 121/198 2024-04-13T03:20:07.098 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:07.098 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-13T03:20:07.098 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:07.264 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-13T03:20:07.341 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-13T03:20:07.399 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-13T03:20:07.471 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-13T03:20:07.507 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-13T03:20:07.544 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-13T03:20:07.592 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-13T03:20:07.627 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-13T03:20:07.670 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-13T03:20:07.711 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-13T03:20:07.848 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-13T03:20:07.909 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-13T03:20:07.939 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-13T03:20:07.980 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-13T03:20:08.075 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-13T03:20:08.105 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-13T03:20:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-13T03:20:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-13T03:20:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-13T03:20:08.269 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-13T03:20:08.397 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-13T03:20:08.445 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-13T03:20:08.473 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-04-13T03:20:08.507 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 82/198 2024-04-13T03:20:08.578 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : uuid-1.6.2-55.el9.x86_64 83/198 2024-04-13T03:20:08.592 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 122/198 2024-04-13T03:20:08.618 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 122/198 2024-04-13T03:20:08.618 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:08.618 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-13T03:20:08.618 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T03:20:08.618 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T03:20:08.619 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:08.619 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 84/198 2024-04-13T03:20:08.649 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 85/198 2024-04-13T03:20:08.684 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 86/198 2024-04-13T03:20:08.717 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 87/198 2024-04-13T03:20:08.758 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 88/198 2024-04-13T03:20:08.793 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 89/198 2024-04-13T03:20:08.851 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 123/198 2024-04-13T03:20:08.874 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 123/198 2024-04-13T03:20:08.874 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:08.874 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-13T03:20:08.874 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T03:20:08.874 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T03:20:08.874 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:08.880 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 90/198 2024-04-13T03:20:08.932 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 91/198 2024-04-13T03:20:08.963 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 92/198 2024-04-13T03:20:09.011 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 93/198 2024-04-13T03:20:09.108 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 94/198 2024-04-13T03:20:09.156 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 95/198 2024-04-13T03:20:09.318 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 124/198 2024-04-13T03:20:09.341 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 124/198 2024-04-13T03:20:09.341 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:09.341 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-13T03:20:09.341 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T03:20:09.341 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T03:20:09.342 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:09.457 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : automake-1.16.2-8.el9.noarch 125/198 2024-04-13T03:20:09.514 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ply-3.11-14.el9.noarch 126/198 2024-04-13T03:20:09.569 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 127/198 2024-04-13T03:20:09.754 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 128/198 2024-04-13T03:20:09.803 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 129/198 2024-04-13T03:20:09.880 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 130/198 2024-04-13T03:20:09.885 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 96/198 2024-04-13T03:20:09.927 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 97/198 2024-04-13T03:20:09.961 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 98/198 2024-04-13T03:20:09.970 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 131/198 2024-04-13T03:20:10.060 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 99/198 2024-04-13T03:20:10.108 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 132/198 2024-04-13T03:20:10.110 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 100/198 2024-04-13T03:20:10.152 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 133/198 2024-04-13T03:20:10.186 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 134/198 2024-04-13T03:20:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 101/198 2024-04-13T03:20:10.217 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 135/198 2024-04-13T03:20:10.238 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 102/198 2024-04-13T03:20:10.252 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-16.el9.x86_64 136/198 2024-04-13T03:20:10.280 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.9-15.el9.x86_64 137/198 2024-04-13T03:20:10.417 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.21-2.el9.x86_64 103/198 2024-04-13T03:20:10.444 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 104/198 2024-04-13T03:20:10.522 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 105/198 2024-04-13T03:20:10.726 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:19.0.0-2884.g218a6914.el9.noarch 138/198 2024-04-13T03:20:10.756 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:19.0.0-2884.g218a6914.el9.noarch 138/198 2024-04-13T03:20:10.788 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2884.g218a6914.e 139/198 2024-04-13T03:20:10.849 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2884.g218a6914. 140/198 2024-04-13T03:20:10.879 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noa 141/198 2024-04-13T03:20:11.159 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 106/198 2024-04-13T03:20:11.284 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 107/198 2024-04-13T03:20:11.908 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noa 141/198 2024-04-13T03:20:11.947 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.n 142/198 2024-04-13T03:20:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 108/198 2024-04-13T03:20:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 109/198 2024-04-13T03:20:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 110/198 2024-04-13T03:20:12.904 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.n 142/198 2024-04-13T03:20:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 111/198 2024-04-13T03:20:12.931 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2884.g218 143/198 2024-04-13T03:20:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 112/198 2024-04-13T03:20:13.031 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2884.g218 143/198 2024-04-13T03:20:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 112/198 2024-04-13T03:20:13.068 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.0.0-2884.g218a6914.el9.x86_64 113/198 2024-04-13T03:20:13.088 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-13T03:20:13.103 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 114/198 2024-04-13T03:20:13.189 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-13T03:20:13.194 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2884.g218a6914.el 144/198 2024-04-13T03:20:13.225 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 145/198 2024-04-13T03:20:13.239 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-13T03:20:13.248 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 145/198 2024-04-13T03:20:13.248 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:13.248 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-13T03:20:13.248 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T03:20:13.249 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T03:20:13.249 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:13.276 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-13T03:20:13.292 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 146/198 2024-04-13T03:20:13.304 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 146/198 2024-04-13T03:20:13.323 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-13T03:20:13.401 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:19.0.0-2884.g218a6914.el9.x86_64 147/198 2024-04-13T03:20:13.419 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-13T03:20:13.532 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-13T03:20:13.573 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-13T03:20:13.596 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-13T03:20:13.607 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 116/198 2024-04-13T03:20:13.634 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-13T03:20:13.662 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 116/198 2024-04-13T03:20:13.684 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.0.0-2884.g218a6914.el9.x86_64 117/198 2024-04-13T03:20:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-13T03:20:14.171 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-13T03:20:14.223 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.0.0-2884.g218a6914.el9.x86_64 10/198 2024-04-13T03:20:14.388 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.0.0-2884.g218a6914.el9.x86_64 10/198 2024-04-13T03:20:14.424 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 11/198 2024-04-13T03:20:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 11/198 2024-04-13T03:20:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.0.0-2884.g218a6914.el9.x86_64 12/198 2024-04-13T03:20:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-13T03:20:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-13T03:20:15.096 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-13T03:20:15.121 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 16/198 2024-04-13T03:20:15.295 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 16/198 2024-04-13T03:20:15.331 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86 17/198 2024-04-13T03:20:15.448 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86 17/198 2024-04-13T03:20:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-13T03:20:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-13T03:20:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-13T03:20:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-13T03:20:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-13T03:20:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 118/198 2024-04-13T03:20:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-13T03:20:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 118/198 2024-04-13T03:20:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-13T03:20:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-13T03:20:16.550 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 118/198 2024-04-13T03:20:16.584 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 119/198 2024-04-13T03:20:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-13T03:20:16.665 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 119/198 2024-04-13T03:20:16.665 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-13T03:20:16.665 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-13T03:20:16.665 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:16.693 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 120/198 2024-04-13T03:20:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-13T03:20:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-13T03:20:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-13T03:20:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-13T03:20:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-13T03:20:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-13T03:20:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-13T03:20:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/198 2024-04-13T03:20:17.884 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 148/198 2024-04-13T03:20:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/198 2024-04-13T03:20:17.944 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:19.0.0-2884.g218a6914.el9.x86_64 149/198 2024-04-13T03:20:17.978 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/198 2024-04-13T03:20:18.027 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 32/198 2024-04-13T03:20:18.073 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 33/198 2024-04-13T03:20:18.201 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 34/198 2024-04-13T03:20:18.286 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 35/198 2024-04-13T03:20:18.325 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 36/198 2024-04-13T03:20:18.388 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 37/198 2024-04-13T03:20:18.510 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 38/198 2024-04-13T03:20:18.567 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.0.0-2884.g218a6914.el9. 39/198 2024-04-13T03:20:18.625 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.0.0-2884.g218a6914.el 40/198 2024-04-13T03:20:18.670 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.0.0-2884.g218a6914.el9.x86_6 41/198 2024-04-13T03:20:18.809 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-13T03:20:18.840 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libtool-2.4.6-45.el9.x86_64 150/198 2024-04-13T03:20:18.885 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 151/198 2024-04-13T03:20:18.886 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-13T03:20:18.907 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 151/198 2024-04-13T03:20:18.908 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:18.908 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-13T03:20:18.908 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T03:20:18.908 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T03:20:18.908 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:18.951 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2884.g218a6 152/198 2024-04-13T03:20:18.972 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2884.g218a6 152/198 2024-04-13T03:20:18.972 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:18.973 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-13T03:20:18.973 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:19.009 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-13T03:20:19.017 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 153/198 2024-04-13T03:20:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 153/198 2024-04-13T03:20:19.040 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:19.040 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-13T03:20:19.040 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-13T03:20:19.040 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-13T03:20:19.040 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:19.201 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-13T03:20:19.295 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-13T03:20:19.343 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 154/198 2024-04-13T03:20:19.366 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 154/198 2024-04-13T03:20:19.366 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:19.366 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-13T03:20:19.366 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T03:20:19.366 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T03:20:19.366 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:19.426 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 155/198 2024-04-13T03:20:19.456 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:19.0.0-2884.g218a6914.el9.x86_64 156/198 2024-04-13T03:20:19.483 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/198 2024-04-13T03:20:19.662 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-shell-2:19.0.0-2884.g218a6914.el9.x86_64 158/198 2024-04-13T03:20:19.714 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 159/198 2024-04-13T03:20:19.753 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-13T03:20:19.806 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-13T03:20:19.850 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-13T03:20:19.874 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:19.0.0-2884.g218a6914.el9.x86_ 163/198 2024-04-13T03:20:19.906 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-13T03:20:19.936 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-13T03:20:19.998 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-13T03:20:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-13T03:20:20.059 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-13T03:20:20.093 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-13T03:20:20.161 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-13T03:20:20.296 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-13T03:20:20.367 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-13T03:20:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-13T03:20:20.442 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-13T03:20:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-13T03:20:20.569 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.0.0-2884.g218a6914.el9.x86_6 53/198 2024-04-13T03:20:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-13T03:20:20.637 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-13T03:20:20.675 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-13T03:20:20.694 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-13T03:20:21.069 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-13T03:20:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-13T03:20:21.372 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-13T03:20:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-13T03:20:21.505 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-top-2:19.0.0-2884.g218a6914.el9.noarch 170/198 2024-04-13T03:20:21.595 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-13T03:20:21.647 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-13T03:20:21.674 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flex-2.6.4-9.el9.x86_64 172/198 2024-04-13T03:20:21.695 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-13T03:20:21.706 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : bison-3.7.4-5.el9.x86_64 173/198 2024-04-13T03:20:21.732 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 174/198 2024-04-13T03:20:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-13T03:20:21.775 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-13T03:20:21.782 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-13T03:20:21.809 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-13T03:20:21.810 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-13T03:20:21.869 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-13T03:20:21.897 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-13T03:20:21.920 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-13T03:20:21.935 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-13T03:20:21.935 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-13T03:20:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-13T03:20:22.012 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-13T03:20:22.038 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-13T03:20:22.058 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-13T03:20:22.071 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-13T03:20:22.097 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-13T03:20:22.102 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-13T03:20:22.121 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-13T03:20:22.154 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-13T03:20:22.166 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-13T03:20:22.194 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-13T03:20:22.226 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-13T03:20:22.232 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-13T03:20:22.252 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-13T03:20:22.277 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-13T03:20:22.302 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-13T03:20:22.328 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-13T03:20:22.334 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-13T03:20:22.335 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-13T03:20:22.479 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-13T03:20:22.514 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-13T03:20:22.514 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-13T03:20:22.541 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-13T03:20:22.581 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-13T03:20:22.630 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-13T03:20:22.655 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-13T03:20:22.686 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-13T03:20:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-13T03:20:22.714 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-13T03:20:22.748 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-13T03:20:22.814 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-13T03:20:22.851 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-13T03:20:22.893 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-13T03:20:22.940 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-13T03:20:23.002 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-13T03:20:23.035 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-13T03:20:23.221 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-13T03:20:23.413 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-13T03:20:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-13T03:20:23.497 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-04-13T03:20:23.542 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 82/198 2024-04-13T03:20:23.598 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-55.el9.x86_64 83/198 2024-04-13T03:20:23.687 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 84/198 2024-04-13T03:20:23.737 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 85/198 2024-04-13T03:20:23.785 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 86/198 2024-04-13T03:20:23.834 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 87/198 2024-04-13T03:20:23.913 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 88/198 2024-04-13T03:20:23.957 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 89/198 2024-04-13T03:20:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 90/198 2024-04-13T03:20:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 91/198 2024-04-13T03:20:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 92/198 2024-04-13T03:20:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 93/198 2024-04-13T03:20:24.495 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 94/198 2024-04-13T03:20:24.566 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 95/198 2024-04-13T03:20:25.590 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 96/198 2024-04-13T03:20:25.647 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 97/198 2024-04-13T03:20:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 98/198 2024-04-13T03:20:26.053 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 99/198 2024-04-13T03:20:26.413 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 100/198 2024-04-13T03:20:26.549 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 101/198 2024-04-13T03:20:26.600 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 102/198 2024-04-13T03:20:26.870 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.21-2.el9.x86_64 103/198 2024-04-13T03:20:26.913 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 104/198 2024-04-13T03:20:27.023 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 105/198 2024-04-13T03:20:27.931 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 106/198 2024-04-13T03:20:28.083 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 107/198 2024-04-13T03:20:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-13T03:20:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:19.0.0-2884.g218a6914.el9.x86_64 1/198 2024-04-13T03:20:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 2/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 3/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 4/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2884.g218a6 5/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 6/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 7/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 8/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 9/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 10/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 11/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:19.0.0-2884.g218a6914.el9.x86_64 12/198 2024-04-13T03:20:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 13/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-shell-2:19.0.0-2884.g218a6914.el9.x86_64 14/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:19.0.0-2884.g218a6914.el9.x86_ 15/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 16/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 17/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:19.0.0-2884.g218a6914.el9.x86_6 18/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86 19/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 20/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2884.g218a6914.el 21/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:19.0.0-2884.g218a6914.el9. 22/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:19.0.0-2884.g218a6914.el9.x86_6 23/198 2024-04-13T03:20:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:19.0.0-2884.g218a6914.el9.x86_64 24/198 2024-04-13T03:20:28.376 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:19.0.0-2884.g218a6914.el9.x86_64 25/198 2024-04-13T03:20:28.376 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:19.0.0-2884.g218a6914.el9.x86_64 26/198 2024-04-13T03:20:28.376 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 27/198 2024-04-13T03:20:28.376 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 28/198 2024-04-13T03:20:28.376 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:19.0.0-2884.g218a6914.el9.x86_64 29/198 2024-04-13T03:20:28.393 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2884.g218a6914. 30/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noa 31/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.n 32/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2884.g218 33/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2884.g218a6914.el 34/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 35/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2884.g218a6914.e 36/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 37/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:19.0.0-2884.g218a6914.el9.noarch 38/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-top-2:19.0.0-2884.g218a6914.el9.noarch 39/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-16.el9.x86_64 42/198 2024-04-13T03:20:28.394 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/198 2024-04-13T03:20:28.395 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 64/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 70/198 2024-04-13T03:20:28.396 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-13.14-1.el9.x86_64 71/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 72/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 73/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 74/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/198 2024-04-13T03:20:28.397 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/198 2024-04-13T03:20:28.398 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-13T03:20:28.399 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-13T03:20:28.400 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-13T03:20:28.401 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-13T03:20:28.402 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:19.0.0-2884.g218a6914.el9.x86_64 151/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 153/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-13T03:20:28.403 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/198 2024-04-13T03:20:28.404 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 173/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 174/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 175/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 176/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 177/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 178/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 179/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 180/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 181/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 182/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 183/198 2024-04-13T03:20:28.405 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 184/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 185/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 186/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 187/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 188/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 189/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 190/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 191/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 192/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 193/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 194/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-13T03:20:28.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-13T03:20:28.407 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-13T03:20:28.427 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 120/198 2024-04-13T03:20:28.428 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-04-13T03:20:28.428 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-04-13T03:20:28.428 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-04-13T03:20:28.428 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-04-13T03:20:28.428 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-04-13T03:20:28.428 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-04-13T03:20:28.428 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-04-13T03:20:28.428 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:28.588 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 121/198 2024-04-13T03:20:28.603 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 121/198 2024-04-13T03:20:28.603 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:28.604 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-13T03:20:28.604 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:29.731 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 108/198 2024-04-13T03:20:29.871 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 109/198 2024-04-13T03:20:29.931 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 110/198 2024-04-13T03:20:30.074 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 122/198 2024-04-13T03:20:30.106 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 122/198 2024-04-13T03:20:30.106 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:30.106 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-13T03:20:30.106 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T03:20:30.107 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T03:20:30.107 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:30.317 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 111/198 2024-04-13T03:20:30.342 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 123/198 2024-04-13T03:20:30.348 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 112/198 2024-04-13T03:20:30.368 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 123/198 2024-04-13T03:20:30.368 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:30.368 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-13T03:20:30.368 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T03:20:30.368 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T03:20:30.368 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:30.470 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 112/198 2024-04-13T03:20:30.505 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.0.0-2884.g218a6914.el9.x86_64 113/198 2024-04-13T03:20:30.568 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 114/198 2024-04-13T03:20:30.819 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 124/198 2024-04-13T03:20:30.841 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 124/198 2024-04-13T03:20:30.841 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:30.842 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-13T03:20:30.842 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T03:20:30.842 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T03:20:30.842 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:30.949 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : automake-1.16.2-8.el9.noarch 125/198 2024-04-13T03:20:31.006 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 126/198 2024-04-13T03:20:31.061 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 127/198 2024-04-13T03:20:31.136 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-13T03:20:31.185 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 116/198 2024-04-13T03:20:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 128/198 2024-04-13T03:20:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 129/198 2024-04-13T03:20:31.312 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 116/198 2024-04-13T03:20:31.323 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-13T03:20:31.323 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:31.323 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-04-13T03:20:31.323 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-13T03:20:31.323 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-13T03:20:31.323 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-13T03:20:31.323 INFO:teuthology.orchestra.run.smithi188.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-13T03:20:31.323 INFO:teuthology.orchestra.run.smithi188.stdout: acl-2.3.1-4.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-13T03:20:31.324 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: automake-1.16.2-8.el9.noarch 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: bison-3.7.4-5.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.325 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.326 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-shell-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: flex-2.6.4-9.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-13T03:20:31.327 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: jq-1.6-16.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-13T03:20:31.328 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: lua-5.4.4-4.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-13T03:20:31.329 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-13.14-1.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-13T03:20:31.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-13T03:20:31.331 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.331 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.331 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.331 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-13T03:20:31.331 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-13T03:20:31.331 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-13T03:20:31.331 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-13T03:20:31.332 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-13T03:20:31.333 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply-3.11-14.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-13T03:20:31.334 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-13T03:20:31.335 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: re2-1:20211101-3.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-13T03:20:31.336 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-13T03:20:31.337 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-13T03:20:31.337 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-13T03:20:31.337 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:20:31.337 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-13T03:20:31.346 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.0.0-2884.g218a6914.el9.x86_64 117/198 2024-04-13T03:20:31.390 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 130/198 2024-04-13T03:20:31.478 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 131/198 2024-04-13T03:20:31.611 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 132/198 2024-04-13T03:20:31.652 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 133/198 2024-04-13T03:20:31.686 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 134/198 2024-04-13T03:20:31.717 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 135/198 2024-04-13T03:20:31.752 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-16.el9.x86_64 136/198 2024-04-13T03:20:31.780 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-15.el9.x86_64 137/198 2024-04-13T03:20:31.792 DEBUG:teuthology.parallel:result is None 2024-04-13T03:20:32.240 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-2884.g218a6914.el9.noarch 138/198 2024-04-13T03:20:32.270 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.0.0-2884.g218a6914.el9.noarch 138/198 2024-04-13T03:20:32.302 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2884.g218a6914.e 139/198 2024-04-13T03:20:32.363 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2884.g218a6914. 140/198 2024-04-13T03:20:32.394 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noa 141/198 2024-04-13T03:20:33.435 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noa 141/198 2024-04-13T03:20:33.470 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.n 142/198 2024-04-13T03:20:34.209 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 118/198 2024-04-13T03:20:34.251 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 118/198 2024-04-13T03:20:34.437 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.n 142/198 2024-04-13T03:20:34.462 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2884.g218 143/198 2024-04-13T03:20:34.562 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2884.g218 143/198 2024-04-13T03:20:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2884.g218a6914.el 144/198 2024-04-13T03:20:34.755 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 145/198 2024-04-13T03:20:34.778 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 145/198 2024-04-13T03:20:34.778 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:34.778 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-13T03:20:34.779 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T03:20:34.779 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T03:20:34.779 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:34.824 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 146/198 2024-04-13T03:20:34.834 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 146/198 2024-04-13T03:20:34.926 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 118/198 2024-04-13T03:20:34.931 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.0.0-2884.g218a6914.el9.x86_64 147/198 2024-04-13T03:20:34.971 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 119/198 2024-04-13T03:20:35.188 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 119/198 2024-04-13T03:20:35.188 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-13T03:20:35.188 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-13T03:20:35.188 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:20:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 120/198 2024-04-13T03:20:39.427 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 148/198 2024-04-13T03:20:39.501 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.0.0-2884.g218a6914.el9.x86_64 149/198 2024-04-13T03:20:40.384 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libtool-2.4.6-45.el9.x86_64 150/198 2024-04-13T03:20:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 151/198 2024-04-13T03:20:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 151/198 2024-04-13T03:20:40.444 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:40.444 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-13T03:20:40.444 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T03:20:40.444 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T03:20:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:40.497 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2884.g218a6 152/198 2024-04-13T03:20:40.519 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2884.g218a6 152/198 2024-04-13T03:20:40.519 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:40.519 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-13T03:20:40.519 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:40.565 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 153/198 2024-04-13T03:20:40.587 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 153/198 2024-04-13T03:20:40.588 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:40.588 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-13T03:20:40.588 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-13T03:20:40.588 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-13T03:20:40.588 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 154/198 2024-04-13T03:20:40.912 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 154/198 2024-04-13T03:20:40.912 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:40.913 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-13T03:20:40.913 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T03:20:40.913 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T03:20:40.913 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:40.964 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 155/198 2024-04-13T03:20:40.995 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.0.0-2884.g218a6914.el9.x86_64 156/198 2024-04-13T03:20:41.022 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/198 2024-04-13T03:20:41.201 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-shell-2:19.0.0-2884.g218a6914.el9.x86_64 158/198 2024-04-13T03:20:41.253 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 159/198 2024-04-13T03:20:41.291 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-13T03:20:41.344 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-13T03:20:41.379 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-13T03:20:41.403 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.0.0-2884.g218a6914.el9.x86_ 163/198 2024-04-13T03:20:41.436 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-13T03:20:41.465 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-13T03:20:41.519 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-13T03:20:41.579 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-13T03:20:41.622 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-13T03:20:42.175 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-13T03:20:42.212 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-13T03:20:42.231 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-13T03:20:43.048 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-top-2:19.0.0-2884.g218a6914.el9.noarch 170/198 2024-04-13T03:20:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-13T03:20:43.228 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flex-2.6.4-9.el9.x86_64 172/198 2024-04-13T03:20:43.276 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : bison-3.7.4-5.el9.x86_64 173/198 2024-04-13T03:20:43.301 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 174/198 2024-04-13T03:20:43.327 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-13T03:20:43.352 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-13T03:20:43.379 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-13T03:20:43.379 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-13T03:20:43.466 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-13T03:20:43.496 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-13T03:20:43.496 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-13T03:20:43.580 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-13T03:20:43.607 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-13T03:20:43.640 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-13T03:20:43.665 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-13T03:20:43.690 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-13T03:20:43.731 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-13T03:20:43.780 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-13T03:20:43.812 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-13T03:20:43.837 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-13T03:20:43.882 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-13T03:20:43.904 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-13T03:20:43.930 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-13T03:20:43.936 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-13T03:20:43.936 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-13T03:20:44.116 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-13T03:20:44.116 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-13T03:20:44.198 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-13T03:20:44.223 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-13T03:20:44.255 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-13T03:20:44.283 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-13T03:20:44.308 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-13T03:20:44.380 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-13T03:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 120/198 2024-04-13T03:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-04-13T03:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-04-13T03:20:48.655 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-04-13T03:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-04-13T03:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-04-13T03:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-04-13T03:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-04-13T03:20:48.656 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:20:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 121/198 2024-04-13T03:20:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 121/198 2024-04-13T03:20:48.959 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:48.959 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-13T03:20:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:20:49.936 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.0.0-2884.g218a6914.el9.x86_64 1/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 2/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 3/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 4/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2884.g218a6 5/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 6/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 7/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 8/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 9/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 10/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 11/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.0.0-2884.g218a6914.el9.x86_64 12/198 2024-04-13T03:20:49.937 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 13/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-shell-2:19.0.0-2884.g218a6914.el9.x86_64 14/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.0.0-2884.g218a6914.el9.x86_ 15/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 16/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 17/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.0.0-2884.g218a6914.el9.x86_6 18/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86 19/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 20/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2884.g218a6914.el 21/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.0.0-2884.g218a6914.el9. 22/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.0.0-2884.g218a6914.el9.x86_6 23/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.0.0-2884.g218a6914.el9.x86_64 24/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.0.0-2884.g218a6914.el9.x86_64 25/198 2024-04-13T03:20:49.938 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.0.0-2884.g218a6914.el9.x86_64 26/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 27/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 28/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.0.0-2884.g218a6914.el9.x86_64 29/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2884.g218a6914. 30/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noa 31/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.n 32/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2884.g218 33/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2884.g218a6914.el 34/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 35/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2884.g218a6914.e 36/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 37/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.0.0-2884.g218a6914.el9.noarch 38/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-top-2:19.0.0-2884.g218a6914.el9.noarch 39/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/198 2024-04-13T03:20:49.939 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-16.el9.x86_64 42/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/198 2024-04-13T03:20:49.940 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 64/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/198 2024-04-13T03:20:49.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 70/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-13.14-1.el9.x86_64 71/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 72/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 73/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 74/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/198 2024-04-13T03:20:49.942 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/198 2024-04-13T03:20:49.943 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-13T03:20:49.944 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-13T03:20:49.945 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-13T03:20:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.0.0-2884.g218a6914.el9.x86_64 151/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 153/198 2024-04-13T03:20:49.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/198 2024-04-13T03:20:49.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 173/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 174/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 175/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 176/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 177/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 178/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 179/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 180/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 181/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 182/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 183/198 2024-04-13T03:20:49.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 184/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 185/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 186/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 187/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 188/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 189/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 190/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 191/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 192/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 193/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 194/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-13T03:20:49.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-13T03:20:50.644 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 122/198 2024-04-13T03:20:50.740 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 122/198 2024-04-13T03:20:50.741 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:50.741 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-13T03:20:50.741 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T03:20:50.741 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-13T03:20:50.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:20:51.011 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 123/198 2024-04-13T03:20:51.096 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 123/198 2024-04-13T03:20:51.097 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:51.097 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-13T03:20:51.097 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T03:20:51.097 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-13T03:20:51.097 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:20:51.612 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 124/198 2024-04-13T03:20:51.694 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 124/198 2024-04-13T03:20:51.694 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:51.694 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-13T03:20:51.694 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T03:20:51.695 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-13T03:20:51.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:20:51.847 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.2-8.el9.noarch 125/198 2024-04-13T03:20:51.932 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 126/198 2024-04-13T03:20:52.022 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 127/198 2024-04-13T03:20:52.297 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 128/198 2024-04-13T03:20:52.361 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 129/198 2024-04-13T03:20:52.454 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 130/198 2024-04-13T03:20:52.586 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 131/198 2024-04-13T03:20:52.775 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 132/198 2024-04-13T03:20:52.828 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 133/198 2024-04-13T03:20:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 134/198 2024-04-13T03:20:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 135/198 2024-04-13T03:20:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.6-16.el9.x86_64 136/198 2024-04-13T03:20:53.008 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-15.el9.x86_64 137/198 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout: acl-2.3.1-4.el9.x86_64 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-13T03:20:53.053 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-13T03:20:53.054 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: automake-1.16.2-8.el9.noarch 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: bison-3.7.4-5.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.055 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.056 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: flex-2.6.4-9.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-13T03:20:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.6-16.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-13T03:20:53.058 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-13T03:20:53.059 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-13.14-1.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-13T03:20:53.060 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-13T03:20:53.061 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-13T03:20:53.062 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-13T03:20:53.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-13T03:20:53.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-3.el9.x86_64 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-13T03:20:53.065 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-13T03:20:53.066 INFO:teuthology.orchestra.run.smithi119.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-13T03:20:53.066 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-13T03:20:53.066 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-13T03:20:53.066 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-13T03:20:53.066 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-13T03:20:53.066 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-13T03:20:53.541 DEBUG:teuthology.parallel:result is None 2024-04-13T03:20:53.652 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.0.0-2884.g218a6914.el9.noarch 138/198 2024-04-13T03:20:53.696 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.0.0-2884.g218a6914.el9.noarch 138/198 2024-04-13T03:20:53.760 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2884.g218a6914.e 139/198 2024-04-13T03:20:53.879 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2884.g218a6914. 140/198 2024-04-13T03:20:53.921 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noa 141/198 2024-04-13T03:20:55.298 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noa 141/198 2024-04-13T03:20:55.340 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.n 142/198 2024-04-13T03:20:56.505 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.n 142/198 2024-04-13T03:20:56.532 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2884.g218 143/198 2024-04-13T03:20:56.684 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2884.g218 143/198 2024-04-13T03:20:56.871 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2884.g218a6914.el 144/198 2024-04-13T03:20:56.911 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 145/198 2024-04-13T03:20:56.993 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 145/198 2024-04-13T03:20:56.994 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:20:56.994 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-13T03:20:56.994 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T03:20:56.994 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-13T03:20:56.994 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:20:57.060 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 146/198 2024-04-13T03:20:57.100 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 146/198 2024-04-13T03:20:57.249 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.0.0-2884.g218a6914.el9.x86_64 147/198 2024-04-13T03:21:02.478 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 148/198 2024-04-13T03:21:02.561 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.0.0-2884.g218a6914.el9.x86_64 149/198 2024-04-13T03:21:03.566 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-45.el9.x86_64 150/198 2024-04-13T03:21:03.601 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 151/198 2024-04-13T03:21:03.677 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 151/198 2024-04-13T03:21:03.678 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:21:03.678 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-13T03:21:03.678 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T03:21:03.678 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-13T03:21:03.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:21:03.727 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2884.g218a6 152/198 2024-04-13T03:21:03.798 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2884.g218a6 152/198 2024-04-13T03:21:03.798 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:21:03.798 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-13T03:21:03.799 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:21:03.869 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 153/198 2024-04-13T03:21:03.951 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 153/198 2024-04-13T03:21:03.951 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:21:03.951 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-13T03:21:03.951 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-13T03:21:03.951 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-13T03:21:03.951 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:21:04.364 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 154/198 2024-04-13T03:21:04.439 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 154/198 2024-04-13T03:21:04.439 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-13T03:21:04.439 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-13T03:21:04.439 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T03:21:04.439 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-13T03:21:04.440 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:21:04.506 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 155/198 2024-04-13T03:21:04.539 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.0.0-2884.g218a6914.el9.x86_64 156/198 2024-04-13T03:21:04.575 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/198 2024-04-13T03:21:04.878 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-shell-2:19.0.0-2884.g218a6914.el9.x86_64 158/198 2024-04-13T03:21:04.943 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 159/198 2024-04-13T03:21:05.004 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-13T03:21:05.097 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-13T03:21:05.143 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-13T03:21:05.188 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.0.0-2884.g218a6914.el9.x86_ 163/198 2024-04-13T03:21:05.224 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-13T03:21:05.276 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-13T03:21:05.355 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-13T03:21:05.484 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-13T03:21:05.518 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-13T03:21:06.513 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-13T03:21:06.557 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-13T03:21:06.617 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-13T03:21:08.062 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:19.0.0-2884.g218a6914.el9.noarch 170/198 2024-04-13T03:21:08.178 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-13T03:21:08.301 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.4-9.el9.x86_64 172/198 2024-04-13T03:21:08.341 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.7.4-5.el9.x86_64 173/198 2024-04-13T03:21:08.366 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 174/198 2024-04-13T03:21:08.400 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-13T03:21:08.452 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-13T03:21:08.494 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-13T03:21:08.495 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-13T03:21:08.639 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-13T03:21:08.689 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-13T03:21:08.689 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-13T03:21:08.860 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-13T03:21:08.905 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-13T03:21:08.951 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-13T03:21:08.992 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-13T03:21:09.026 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-13T03:21:09.098 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-13T03:21:09.173 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-13T03:21:09.221 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-13T03:21:09.254 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-13T03:21:09.288 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-13T03:21:09.321 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-13T03:21:09.355 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-13T03:21:09.393 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-13T03:21:09.393 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-13T03:21:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-13T03:21:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-13T03:21:09.869 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-13T03:21:09.900 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-13T03:21:09.954 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-13T03:21:09.987 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-13T03:21:10.043 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-13T03:21:10.167 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-13T03:21:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-13T03:21:10.442 INFO:teuthology.orchestra.run.smithi016.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-13T03:21:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.0.0-2884.g218a6914.el9.x86_64 1/198 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 2/198 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 3/198 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 4/198 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2884.g218a6 5/198 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 6/198 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 7/198 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 8/198 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 9/198 2024-04-13T03:21:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 10/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 11/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.0.0-2884.g218a6914.el9.x86_64 12/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 13/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-shell-2:19.0.0-2884.g218a6914.el9.x86_64 14/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.0.0-2884.g218a6914.el9.x86_ 15/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 16/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 17/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.0.0-2884.g218a6914.el9.x86_6 18/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86 19/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 20/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2884.g218a6914.el 21/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.0.0-2884.g218a6914.el9. 22/198 2024-04-13T03:21:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.0.0-2884.g218a6914.el9.x86_6 23/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.0.0-2884.g218a6914.el9.x86_64 24/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.0.0-2884.g218a6914.el9.x86_64 25/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.0.0-2884.g218a6914.el9.x86_64 26/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 27/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 28/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.0.0-2884.g218a6914.el9.x86_64 29/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2884.g218a6914. 30/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noa 31/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.n 32/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2884.g218 33/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2884.g218a6914.el 34/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 35/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2884.g218a6914.e 36/198 2024-04-13T03:21:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 37/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.0.0-2884.g218a6914.el9.noarch 38/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:19.0.0-2884.g218a6914.el9.noarch 39/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.6-16.el9.x86_64 42/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/198 2024-04-13T03:21:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/198 2024-04-13T03:21:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 64/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 70/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-13.14-1.el9.x86_64 71/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 72/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 73/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 74/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/198 2024-04-13T03:21:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/198 2024-04-13T03:21:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/198 2024-04-13T03:21:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-13T03:21:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-13T03:21:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-13T03:21:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.0.0-2884.g218a6914.el9.x86_64 151/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 153/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-13T03:21:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/198 2024-04-13T03:21:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 173/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 174/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 175/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 176/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 177/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 178/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 179/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 180/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 181/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 182/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 183/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 184/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 185/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 186/198 2024-04-13T03:21:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 187/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 188/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 189/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 190/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 191/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 192/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 193/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 194/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-13T03:21:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-13T03:21:19.063 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-13T03:21:19.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: acl-2.3.1-4.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-13T03:21:19.064 INFO:teuthology.orchestra.run.smithi016.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-13T03:21:19.065 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-13T03:21:19.066 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.2-8.el9.noarch 2024-04-13T03:21:19.066 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.7.4-5.el9.x86_64 2024-04-13T03:21:19.066 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-13T03:21:19.066 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.066 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.066 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.066 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:19.0.0-2884.g218a6914.el9.noarch 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.4-9.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-13T03:21:19.071 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: jq-1.6-16.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.072 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-13T03:21:19.073 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-13.14-1.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.074 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-13T03:21:19.075 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-13T03:21:19.076 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.077 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-13T03:21:19.078 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.0.0-2884.g218a6914.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-3.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-13T03:21:19.079 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-13T03:21:19.080 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-13T03:21:19.080 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:21:19.080 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-13T03:21:19.745 DEBUG:teuthology.parallel:result is None 2024-04-13T03:21:19.745 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:21:19.985 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-13T03:21:20.062 INFO:teuthology.orchestra.run.smithi016.stdout:19.0.0-2884.g218a6914.el9 2024-04-13T03:21:20.063 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2884.g218a6914.el9 2024-04-13T03:21:20.063 INFO:teuthology.task.install:The correct ceph version 19.0.0-2884.g218a6914 is installed. 2024-04-13T03:21:20.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:21:20.205 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-13T03:21:20.231 INFO:teuthology.orchestra.run.smithi119.stdout:19.0.0-2884.g218a6914.el9 2024-04-13T03:21:20.231 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2884.g218a6914.el9 2024-04-13T03:21:20.231 INFO:teuthology.task.install:The correct ceph version 19.0.0-2884.g218a6914 is installed. 2024-04-13T03:21:20.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:21:20.476 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-13T03:21:20.502 INFO:teuthology.orchestra.run.smithi188.stdout:19.0.0-2884.g218a6914.el9 2024-04-13T03:21:20.502 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2884.g218a6914.el9 2024-04-13T03:21:20.503 INFO:teuthology.task.install:The correct ceph version 19.0.0-2884.g218a6914 is installed. 2024-04-13T03:21:20.505 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-13T03:21:20.505 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:21:20.505 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-13T03:21:20.592 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:21:20.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-13T03:21:20.627 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:21:20.627 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-13T03:21:20.662 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-13T03:21:20.662 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:21:20.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-04-13T03:21:20.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-13T03:21:20.877 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:21:20.877 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-04-13T03:21:20.908 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-13T03:21:20.977 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:21:20.977 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-04-13T03:21:21.005 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-13T03:21:21.072 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-13T03:21:21.072 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:21:21.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-13T03:21:21.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-13T03:21:21.261 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:21:21.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-13T03:21:21.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-13T03:21:21.356 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:21:21.356 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-13T03:21:21.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-13T03:21:21.450 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-13T03:21:21.450 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:21:21.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-04-13T03:21:21.530 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-13T03:21:21.651 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:21:21.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-04-13T03:21:21.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-13T03:21:21.745 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:21:21.745 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-04-13T03:21:21.772 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-13T03:21:21.839 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-13T03:21:21.942 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-13T03:21:21.943 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:21:21.943 INFO:tasks.cephadm:Cluster fsid is e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:21:21.943 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-13T03:21:21.943 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.16', 'mon.b': '172.21.15.119', 'mon.c': '172.21.15.188'} 2024-04-13T03:21:21.944 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-04-13T03:21:21.944 INFO:tasks.cephadm:First mgr is x 2024-04-13T03:21:21.944 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-13T03:21:21.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-04-13T03:21:22.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname $(hostname -s) 2024-04-13T03:21:22.053 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2024-04-13T03:21:22.080 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-13T03:21:22.080 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:21:22.222 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78520/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2884-g218a6914', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2884.g218a6914'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-12 10:42:09.634437', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240412.042930', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/flavors/default/', 'archs': ['arm64', 'source', 'x86_64'], 'distro': 'centos'}] 2024-04-13T03:21:22.491 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-vshankar-testing-20240412.042930, sha1 218a691417ef57c5d22a8d5cf3c5f41f7b542838 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:21:22.493 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/x86_64/flavors/default/cephadm 2024-04-13T03:21:22.493 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-13T03:21:23.137 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 781652 Apr 13 03:21 /home/ubuntu/cephtest/cephadm 2024-04-13T03:21:23.138 DEBUG:teuthology.orchestra.run.smithi119:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-13T03:21:23.725 INFO:teuthology.orchestra.run.smithi119.stdout:-rw-r--r--. 1 ubuntu ubuntu 781652 Apr 13 03:21 /home/ubuntu/cephtest/cephadm 2024-04-13T03:21:23.726 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240412.042930/218a691417ef57c5d22a8d5cf3c5f41f7b542838/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-13T03:21:24.210 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-r--r--. 1 ubuntu ubuntu 781652 Apr 13 03:21 /home/ubuntu/cephtest/cephadm 2024-04-13T03:21:24.210 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-13T03:21:24.260 DEBUG:teuthology.orchestra.run.smithi119:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-13T03:21:24.279 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-13T03:21:24.318 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 on all hosts... 2024-04-13T03:21:24.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 pull 2024-04-13T03:21:24.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 pull 2024-04-13T03:21:24.329 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 pull 2024-04-13T03:21:25.178 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838... 2024-04-13T03:21:25.222 INFO:teuthology.orchestra.run.smithi119.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838... 2024-04-13T03:21:25.936 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838... 2024-04-13T03:22:05.034 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-13T03:22:05.034 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)", 2024-04-13T03:22:05.034 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05", 2024-04-13T03:22:05.034 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2024-04-13T03:22:05.034 INFO:teuthology.orchestra.run.smithi188.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa" 2024-04-13T03:22:05.035 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-13T03:22:05.035 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-13T03:22:05.134 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-13T03:22:05.134 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)", 2024-04-13T03:22:05.134 INFO:teuthology.orchestra.run.smithi119.stdout: "image_id": "a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05", 2024-04-13T03:22:05.134 INFO:teuthology.orchestra.run.smithi119.stdout: "repo_digests": [ 2024-04-13T03:22:05.134 INFO:teuthology.orchestra.run.smithi119.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa" 2024-04-13T03:22:05.134 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-04-13T03:22:05.134 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-13T03:22:09.346 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:22:09.347 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)", 2024-04-13T03:22:09.347 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05", 2024-04-13T03:22:09.347 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-04-13T03:22:09.347 INFO:teuthology.orchestra.run.smithi016.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa" 2024-04-13T03:22:09.347 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:22:09.347 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:22:09.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-04-13T03:22:09.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2024-04-13T03:22:09.551 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2024-04-13T03:22:09.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-04-13T03:22:09.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /etc/ceph 2024-04-13T03:22:09.725 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2024-04-13T03:22:09.757 INFO:tasks.cephadm:Writing seed config... 2024-04-13T03:22:09.758 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-13T03:22:09.758 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-13T03:22:09.758 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-13T03:22:09.758 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-13T03:22:09.758 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-13T03:22:09.758 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-13T03:22:09.758 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-13T03:22:09.758 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-13T03:22:09.759 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-13T03:22:09.760 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-13T03:22:09.761 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-13T03:22:09.761 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-13T03:22:09.761 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-13T03:22:09.761 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-13T03:22:09.761 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-13T03:22:09.761 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-13T03:22:09.761 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-13T03:22:09.761 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-13T03:22:09.762 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:22:09.762 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-13T03:22:09.821 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e6bf0314-f944-11ee-bc8d-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-13T03:22:09.822 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.a.service 2024-04-13T03:22:09.870 DEBUG:teuthology.orchestra.run.smithi016:mgr.x> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.x.service 2024-04-13T03:22:09.916 INFO:tasks.cephadm:Bootstrapping... 2024-04-13T03:22:09.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 -v bootstrap --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-13T03:22:10.198 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-13T03:22:10.198 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838', '-v', 'bootstrap', '--fsid', 'e6bf0314-f944-11ee-bc8d-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-04-13T03:22:10.199 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-13T03:22:10.199 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-04-13T03:22:10.249 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-13T03:22:10.250 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-04-13T03:22:10.250 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-04-13T03:22:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-13T03:22:10.265 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-13T03:22:10.279 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-13T03:22:10.279 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-13T03:22:10.295 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-04-13T03:22:10.311 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-04-13T03:22:10.312 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-04-13T03:22:10.312 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-04-13T03:22:10.365 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-13T03:22:10.366 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-13T03:22:10.366 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-04-13T03:22:10.366 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-04-13T03:22:10.384 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-13T03:22:10.384 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-13T03:22:10.407 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-13T03:22:10.407 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-13T03:22:10.429 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-04-13T03:22:10.449 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-04-13T03:22:10.449 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-04-13T03:22:10.449 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-04-13T03:22:10.449 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:10.449 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 140637160600480 on /run/cephadm/e6bf0314-f944-11ee-bc8d-c7b262605968.lock 2024-04-13T03:22:10.450 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 140637160600480 acquired on /run/cephadm/e6bf0314-f944-11ee-bc8d-c7b262605968.lock 2024-04-13T03:22:10.451 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-04-13T03:22:10.452 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-04-13T03:22:10.452 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-13T03:22:10.465 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-04-13T03:22:10.465 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-04-13T03:22:10.476 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-13T03:22:10.476 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-13T03:22:10.476 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-13T03:22:10.487 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-13T03:22:10.487 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-13T03:22:10.487 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-13T03:22:10.487 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-13T03:22:10.488 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-13T03:22:10.488 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-13T03:22:10.490 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-04-13T03:22:10.490 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-04-13T03:22:10.490 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-13T03:22:10.493 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-13T03:22:10.495 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838... 2024-04-13T03:22:11.842 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 2024-04-13T03:22:11.842 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838... 2024-04-13T03:22:11.842 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-04-13T03:22:11.843 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:bc174130f2d13ad2c3f0a0364f095cc7358abb3528a6563dc6f800a0c20467de 2024-04-13T03:22:11.843 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:9793b5b5582798bde394f4383d142e9f248a25e4ebfd72ee05ea9ec6251a877d 2024-04-13T03:22:11.843 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 2024-04-13T03:22:11.843 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-13T03:22:13.158 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev) 2024-04-13T03:22:13.159 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev) 2024-04-13T03:22:13.159 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-04-13T03:22:14.328 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-04-13T03:22:14.328 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-04-13T03:22:15.466 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQBm+hlmg6yKLxAAveuA9WkWc71Sjyqp7UVH9g== 2024-04-13T03:22:16.607 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQBn+hlmyyUwORAABalD1s9xK7QzGlp9LLJlNQ== 2024-04-13T03:22:17.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQBp+hlm8/pAAxAAoJkc5QOYNJ3eu0wVnBaGxg== 2024-04-13T03:22:17.822 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for a [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:22:18.969 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-04-13T03:22:20.234 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.a on 2024-04-13T03:22:20.664 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-13T03:22:20.898 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-13T03:22:21.200 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968.target → /etc/systemd/system/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968.target. 2024-04-13T03:22:21.200 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968.target → /etc/systemd/system/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968.target. 2024-04-13T03:22:21.590 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.a 2024-04-13T03:22:21.590 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.a.service: Unit ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.a.service not loaded. 2024-04-13T03:22:21.878 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968.target.wants/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.a.service → /etc/systemd/system/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@.service. 2024-04-13T03:22:22.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:22 smithi016 podman[31386]: 2024-04-13 03:22:22.278814092 +0000 UTC m=+0.138643385 container create 97e987feaaeab22c60101bd5b15e6c647c8b5414b771aed81c2483cc007ad19c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:22:22.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:22 smithi016 podman[31386]: 2024-04-13 03:22:22.566223253 +0000 UTC m=+0.426052589 container init 97e987feaaeab22c60101bd5b15e6c647c8b5414b771aed81c2483cc007ad19c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-13T03:22:22.688 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-13T03:22:22.688 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-13T03:22:22.703 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-13T03:22:22.703 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-13T03:22:22.703 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-13T03:22:22.703 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-04-13T03:22:22.704 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-04-13T03:22:22.704 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-04-13T03:22:22.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:22 smithi016 podman[31386]: 2024-04-13 03:22:22.578456872 +0000 UTC m=+0.438286168 container start 97e987feaaeab22c60101bd5b15e6c647c8b5414b771aed81c2483cc007ad19c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:22:22.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:22 smithi016 bash[31386]: 97e987feaaeab22c60101bd5b15e6c647c8b5414b771aed81c2483cc007ad19c 2024-04-13T03:22:22.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:22 smithi016 ceph-mon[31405]: mkfs e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:22.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:22 smithi016 systemd[1]: Started Ceph mon.a for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:22:22.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:22 smithi016 ceph-mon[31405]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-13T03:22:23.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-13T03:22:23.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: monmap epoch 1 2024-04-13T03:22:23.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:23.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: last_changed 2024-04-13T03:22:18.286080+0000 2024-04-13T03:22:23.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:22:23.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: min_mon_release 19 (squid) 2024-04-13T03:22:23.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: election_strategy: 1 2024-04-13T03:22:23.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:22:23.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: fsmap 2024-04-13T03:22:23.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: osdmap e1: 0 total, 0 up, 0 in 2024-04-13T03:22:23.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: mgrmap e1: no daemons active 2024-04-13T03:22:23.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:23 smithi016 ceph-mon[31405]: from='client.? 172.21.15.16:0/2336897396' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-13T03:22:24.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-04-13T03:22:24.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:24.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-13T03:22:24.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:24.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-04-13T03:22:24.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.783442s) 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.345+0000 7f578d400700 1 Processor -- start 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.345+0000 7f578d400700 1 -- start start 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.345+0000 7f578d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f578810f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:24.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.345+0000 7f578d400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f578810f960 con 0x7f578810f000 2024-04-13T03:22:24.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.346+0000 7f5786800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f578810f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:24.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.346+0000 7f5786800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f578810f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55612/0 (socket says 172.21.15.16:55612) 2024-04-13T03:22:24.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.346+0000 7f5786800700 1 -- 172.21.15.16:0/886120827 learned_addr learned my addr 172.21.15.16:0/886120827 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:24.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.347+0000 7f5786800700 1 -- 172.21.15.16:0/886120827 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578810fae0 con 0x7f578810f000 2024-04-13T03:22:24.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.347+0000 7f5786800700 1 --2- 172.21.15.16:0/886120827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f578810f420 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f57780098a0 tx=0x7f5778009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bd0cc2bc30e413 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:24.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.348+0000 7f5785400700 1 -- 172.21.15.16:0/886120827 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5778004160 con 0x7f578810f000 2024-04-13T03:22:24.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.348+0000 7f5785400700 1 -- 172.21.15.16:0/886120827 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5778004300 con 0x7f578810f000 2024-04-13T03:22:24.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.348+0000 7f5785400700 1 -- 172.21.15.16:0/886120827 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57780036a0 con 0x7f578810f000 2024-04-13T03:22:24.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.348+0000 7f578d400700 1 -- 172.21.15.16:0/886120827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 msgr2=0x7f578810f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:24.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.348+0000 7f578d400700 1 --2- 172.21.15.16:0/886120827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f578810f420 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f57780098a0 tx=0x7f5778009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:24.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.348+0000 7f578d400700 1 -- 172.21.15.16:0/886120827 shutdown_connections 2024-04-13T03:22:24.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.348+0000 7f578d400700 1 --2- 172.21.15.16:0/886120827 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f578810f420 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:24.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.348+0000 7f578d400700 1 -- 172.21.15.16:0/886120827 >> 172.21.15.16:0/886120827 conn(0x7f578807de00 msgr2=0x7f5788080220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:24.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.349+0000 7f578d400700 1 -- 172.21.15.16:0/886120827 shutdown_connections 2024-04-13T03:22:24.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.349+0000 7f578d400700 1 -- 172.21.15.16:0/886120827 wait complete. 2024-04-13T03:22:24.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.350+0000 7f578d400700 1 Processor -- start 2024-04-13T03:22:24.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.350+0000 7f578d400700 1 -- start start 2024-04-13T03:22:24.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.350+0000 7f578d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f57881a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:24.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.350+0000 7f578d400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57881100d0 con 0x7f578810f000 2024-04-13T03:22:24.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.350+0000 7f5786800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f57881a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:24.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.351+0000 7f5786800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f57881a22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55620/0 (socket says 172.21.15.16:55620) 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.351+0000 7f5786800700 1 -- 172.21.15.16:0/2336897396 learned_addr learned my addr 172.21.15.16:0/2336897396 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.351+0000 7f5786800700 1 -- 172.21.15.16:0/2336897396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57881a2820 con 0x7f578810f000 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.351+0000 7f5786800700 1 --2- 172.21.15.16:0/2336897396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f57881a22e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5778009fb0 tx=0x7f5778004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.352+0000 7f577f400700 1 -- 172.21.15.16:0/2336897396 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5778003de0 con 0x7f578810f000 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.352+0000 7f577f400700 1 -- 172.21.15.16:0/2336897396 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5778011630 con 0x7f578810f000 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.352+0000 7f577f400700 1 -- 172.21.15.16:0/2336897396 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5778011930 con 0x7f578810f000 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.352+0000 7f578d400700 1 -- 172.21.15.16:0/2336897396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57881a2ab0 con 0x7f578810f000 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.352+0000 7f578d400700 1 -- 172.21.15.16:0/2336897396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57881a57a0 con 0x7f578810f000 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.352+0000 7f577f400700 1 -- 172.21.15.16:0/2336897396 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5778011ad0 con 0x7f578810f000 2024-04-13T03:22:24.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.352+0000 7f577f400700 1 -- 172.21.15.16:0/2336897396 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f577801b450 con 0x7f578810f000 2024-04-13T03:22:24.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.353+0000 7f578d400700 1 -- 172.21.15.16:0/2336897396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5788064110 con 0x7f578810f000 2024-04-13T03:22:24.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.356+0000 7f577f400700 1 -- 172.21.15.16:0/2336897396 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5778004260 con 0x7f578810f000 2024-04-13T03:22:24.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.438+0000 7f578d400700 1 -- 172.21.15.16:0/2336897396 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f578802e0a0 con 0x7f578810f000 2024-04-13T03:22:24.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.439+0000 7f577f400700 1 -- 172.21.15.16:0/2336897396 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f5778025480 con 0x7f578810f000 2024-04-13T03:22:24.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.440+0000 7f578d400700 1 -- 172.21.15.16:0/2336897396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 msgr2=0x7f57881a22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:24.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.440+0000 7f578d400700 1 --2- 172.21.15.16:0/2336897396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f57881a22e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5778009fb0 tx=0x7f5778004750 comp rx=0 tx=0).stop 2024-04-13T03:22:24.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.441+0000 7f578d400700 1 -- 172.21.15.16:0/2336897396 shutdown_connections 2024-04-13T03:22:24.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.441+0000 7f578d400700 1 --2- 172.21.15.16:0/2336897396 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f578810f000 0x7f57881a22e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:24.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.441+0000 7f578d400700 1 -- 172.21.15.16:0/2336897396 >> 172.21.15.16:0/2336897396 conn(0x7f578807de00 msgr2=0x7f5788198f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:24.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.441+0000 7f578d400700 1 -- 172.21.15.16:0/2336897396 shutdown_connections 2024-04-13T03:22:24.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:23.441+0000 7f578d400700 1 -- 172.21.15.16:0/2336897396 wait complete. 2024-04-13T03:22:24.159 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-04-13T03:22:24.159 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-04-13T03:22:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:24 smithi016 ceph-mon[31405]: from='client.? 172.21.15.16:0/2441430531' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-13T03:22:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:24 smithi016 ceph-mon[31405]: from='client.? 172.21.15.16:0/2441430531' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-13T03:22:25.526 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:25.526 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-04-13T03:22:25.526 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:25.526 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-13T03:22:25.526 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-13T03:22:25.526 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-13T03:22:25.526 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-13T03:22:25.527 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-13T03:22:25.530 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:25.530 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.770+0000 7f75d4200700 1 Processor -- start 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.770+0000 7f75d4200700 1 -- start start 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.770+0000 7f75d4200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc10acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:25.531 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.770+0000 7f75d4200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc10b2a0 con 0x7f75cc10a8b0 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.771+0000 7f75d1800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc10acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.771+0000 7f75d1800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc10acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55630/0 (socket says 172.21.15.16:55630) 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.771+0000 7f75d1800700 1 -- 172.21.15.16:0/3981935717 learned_addr learned my addr 172.21.15.16:0/3981935717 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.772+0000 7f75d1800700 1 -- 172.21.15.16:0/3981935717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75cc10ba10 con 0x7f75cc10a8b0 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.772+0000 7f75d1800700 1 --2- 172.21.15.16:0/3981935717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc10acd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f75c00098a0 tx=0x7f75c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa0b9b671807c68 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.772+0000 7f75cbe00700 1 -- 172.21.15.16:0/3981935717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c0004160 con 0x7f75cc10a8b0 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.772+0000 7f75cbe00700 1 -- 172.21.15.16:0/3981935717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c0004300 con 0x7f75cc10a8b0 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.772+0000 7f75cbe00700 1 -- 172.21.15.16:0/3981935717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c00036a0 con 0x7f75cc10a8b0 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.773+0000 7f75d4200700 1 -- 172.21.15.16:0/3981935717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 msgr2=0x7f75cc10acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.773+0000 7f75d4200700 1 --2- 172.21.15.16:0/3981935717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc10acd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f75c00098a0 tx=0x7f75c0009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.773+0000 7f75d4200700 1 -- 172.21.15.16:0/3981935717 shutdown_connections 2024-04-13T03:22:25.532 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.773+0000 7f75d4200700 1 --2- 172.21.15.16:0/3981935717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc10acd0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:25.533 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.773+0000 7f75d4200700 1 -- 172.21.15.16:0/3981935717 >> 172.21.15.16:0/3981935717 conn(0x7f75cc07de00 msgr2=0x7f75cc1084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:25.533 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.773+0000 7f75d4200700 1 -- 172.21.15.16:0/3981935717 shutdown_connections 2024-04-13T03:22:25.533 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.773+0000 7f75d4200700 1 -- 172.21.15.16:0/3981935717 wait complete. 2024-04-13T03:22:25.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.774+0000 7f75d4200700 1 Processor -- start 2024-04-13T03:22:25.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.775+0000 7f75d4200700 1 -- start start 2024-04-13T03:22:25.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.775+0000 7f75d4200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:25.536 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.775+0000 7f75d4200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc10bf40 con 0x7f75cc10a8b0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.775+0000 7f75d1800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.775+0000 7f75d1800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc19df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55642/0 (socket says 172.21.15.16:55642) 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.775+0000 7f75d1800700 1 -- 172.21.15.16:0/2441430531 learned_addr learned my addr 172.21.15.16:0/2441430531 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.776+0000 7f75d1800700 1 -- 172.21.15.16:0/2441430531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75cc19e450 con 0x7f75cc10a8b0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.776+0000 7f75d1800700 1 --2- 172.21.15.16:0/2441430531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc19df10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f75c0000c00 tx=0x7f75c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.776+0000 7f75ca000700 1 -- 172.21.15.16:0/2441430531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c0003db0 con 0x7f75cc10a8b0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.776+0000 7f75ca000700 1 -- 172.21.15.16:0/2441430531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c0011630 con 0x7f75cc10a8b0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.776+0000 7f75ca000700 1 -- 172.21.15.16:0/2441430531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c0011950 con 0x7f75cc10a8b0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.776+0000 7f75d4200700 1 -- 172.21.15.16:0/2441430531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75cc19e6e0 con 0x7f75cc10a8b0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.777+0000 7f75d4200700 1 -- 172.21.15.16:0/2441430531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75cc1a13d0 con 0x7f75cc10a8b0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.777+0000 7f75ca000700 1 -- 172.21.15.16:0/2441430531 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f75c0011af0 con 0x7f75cc10a8b0 2024-04-13T03:22:25.537 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.777+0000 7f75ca000700 1 -- 172.21.15.16:0/2441430531 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f75c001c450 con 0x7f75cc10a8b0 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.778+0000 7f75d4200700 1 -- 172.21.15.16:0/2441430531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75cc064110 con 0x7f75cc10a8b0 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.781+0000 7f75ca000700 1 -- 172.21.15.16:0/2441430531 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f75c0032020 con 0x7f75cc10a8b0 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.859+0000 7f75d4200700 1 -- 172.21.15.16:0/2441430531 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f75cc02e120 con 0x7f75cc10a8b0 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.868+0000 7f75ca000700 1 -- 172.21.15.16:0/2441430531 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f75c0004260 con 0x7f75cc10a8b0 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.870+0000 7f75ca000700 1 -- 172.21.15.16:0/2441430531 <== mon.0 v2:172.21.15.16:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75c001cb60 con 0x7f75cc10a8b0 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.872+0000 7f75d4200700 1 -- 172.21.15.16:0/2441430531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 msgr2=0x7f75cc19df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.872+0000 7f75d4200700 1 --2- 172.21.15.16:0/2441430531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc19df10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f75c0000c00 tx=0x7f75c0004750 comp rx=0 tx=0).stop 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.872+0000 7f75d4200700 1 -- 172.21.15.16:0/2441430531 shutdown_connections 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.872+0000 7f75d4200700 1 --2- 172.21.15.16:0/2441430531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75cc10a8b0 0x7f75cc19df10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.872+0000 7f75d4200700 1 -- 172.21.15.16:0/2441430531 >> 172.21.15.16:0/2441430531 conn(0x7f75cc07de00 msgr2=0x7f75cc194c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.872+0000 7f75d4200700 1 -- 172.21.15.16:0/2441430531 shutdown_connections 2024-04-13T03:22:25.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:24.872+0000 7f75d4200700 1 -- 172.21.15.16:0/2441430531 wait complete. 2024-04-13T03:22:25.539 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-04-13T03:22:26.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:26 smithi016 ceph-mon[31405]: from='client.? 172.21.15.16:0/3877489995' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:22:26.881 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.154+0000 7fc3e7200700 1 Processor -- start 2024-04-13T03:22:26.881 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.155+0000 7fc3e7200700 1 -- start start 2024-04-13T03:22:26.881 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.155+0000 7fc3e7200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e8108900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:26.881 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.155+0000 7fc3e7200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e8108ed0 con 0x7fc3e81084e0 2024-04-13T03:22:26.883 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.155+0000 7fc3e5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e8108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:26.883 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.156+0000 7fc3e5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e8108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55648/0 (socket says 172.21.15.16:55648) 2024-04-13T03:22:26.883 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.156+0000 7fc3e5e00700 1 -- 172.21.15.16:0/2616795279 learned_addr learned my addr 172.21.15.16:0/2616795279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:26.883 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.156+0000 7fc3e5e00700 1 -- 172.21.15.16:0/2616795279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e8109640 con 0x7fc3e81084e0 2024-04-13T03:22:26.883 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.156+0000 7fc3e5e00700 1 --2- 172.21.15.16:0/2616795279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e8108900 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80098a0 tx=0x7fc3d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298dfacfaa7bba server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:26.883 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.157+0000 7fc3e5400700 1 -- 172.21.15.16:0/2616795279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d8004160 con 0x7fc3e81084e0 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.157+0000 7fc3e5400700 1 -- 172.21.15.16:0/2616795279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc3d8004300 con 0x7fc3e81084e0 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.157+0000 7fc3e5400700 1 -- 172.21.15.16:0/2616795279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d8011b50 con 0x7fc3e81084e0 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.157+0000 7fc3e7200700 1 -- 172.21.15.16:0/2616795279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 msgr2=0x7fc3e8108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.157+0000 7fc3e7200700 1 --2- 172.21.15.16:0/2616795279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e8108900 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80098a0 tx=0x7fc3d8009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.157+0000 7fc3e7200700 1 -- 172.21.15.16:0/2616795279 shutdown_connections 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.157+0000 7fc3e7200700 1 --2- 172.21.15.16:0/2616795279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e8108900 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.157+0000 7fc3e7200700 1 -- 172.21.15.16:0/2616795279 >> 172.21.15.16:0/2616795279 conn(0x7fc3e8103b20 msgr2=0x7fc3e8105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.158+0000 7fc3e7200700 1 -- 172.21.15.16:0/2616795279 shutdown_connections 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.158+0000 7fc3e7200700 1 -- 172.21.15.16:0/2616795279 wait complete. 2024-04-13T03:22:26.884 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.159+0000 7fc3e7200700 1 Processor -- start 2024-04-13T03:22:26.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.159+0000 7fc3e7200700 1 -- start start 2024-04-13T03:22:26.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.160+0000 7fc3e7200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:26.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.160+0000 7fc3e7200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e810c9f0 con 0x7fc3e81084e0 2024-04-13T03:22:26.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.160+0000 7fc3e5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:26.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.160+0000 7fc3e5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e819c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55660/0 (socket says 172.21.15.16:55660) 2024-04-13T03:22:26.887 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.160+0000 7fc3e5e00700 1 -- 172.21.15.16:0/3877489995 learned_addr learned my addr 172.21.15.16:0/3877489995 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.160+0000 7fc3e5e00700 1 -- 172.21.15.16:0/3877489995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e819c9c0 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.161+0000 7fc3e5e00700 1 --2- 172.21.15.16:0/3877489995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e819c480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8011850 tx=0x7fc3d8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.161+0000 7fc3dea00700 1 -- 172.21.15.16:0/3877489995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d8003af0 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.161+0000 7fc3dea00700 1 -- 172.21.15.16:0/3877489995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc3d8020070 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.161+0000 7fc3e7200700 1 -- 172.21.15.16:0/3877489995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e819cc50 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.161+0000 7fc3dea00700 1 -- 172.21.15.16:0/3877489995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d80195a0 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.161+0000 7fc3e7200700 1 -- 172.21.15.16:0/3877489995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e819f940 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.161+0000 7fc3dea00700 1 -- 172.21.15.16:0/3877489995 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc3d8019be0 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.161+0000 7fc3dea00700 1 -- 172.21.15.16:0/3877489995 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc3d8011e00 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.162+0000 7fc3e7200700 1 -- 172.21.15.16:0/3877489995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e8064110 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.165+0000 7fc3dea00700 1 -- 172.21.15.16:0/3877489995 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc3d8003c90 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.244+0000 7fc3e7200700 1 -- 172.21.15.16:0/3877489995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc3e807c0d0 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.245+0000 7fc3dea00700 1 -- 172.21.15.16:0/3877489995 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fc3e807c0d0 con 0x7fc3e81084e0 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.247+0000 7fc3e7200700 1 -- 172.21.15.16:0/3877489995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 msgr2=0x7fc3e819c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.247+0000 7fc3e7200700 1 --2- 172.21.15.16:0/3877489995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e819c480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8011850 tx=0x7fc3d8011c10 comp rx=0 tx=0).stop 2024-04-13T03:22:26.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.247+0000 7fc3e7200700 1 -- 172.21.15.16:0/3877489995 shutdown_connections 2024-04-13T03:22:26.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.247+0000 7fc3e7200700 1 --2- 172.21.15.16:0/3877489995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e81084e0 0x7fc3e819c480 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:26.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.247+0000 7fc3e7200700 1 -- 172.21.15.16:0/3877489995 >> 172.21.15.16:0/3877489995 conn(0x7fc3e8103b20 msgr2=0x7fc3e8104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:26.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.247+0000 7fc3e7200700 1 -- 172.21.15.16:0/3877489995 shutdown_connections 2024-04-13T03:22:26.889 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:26.247+0000 7fc3e7200700 1 -- 172.21.15.16:0/3877489995 wait complete. 2024-04-13T03:22:26.889 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-04-13T03:22:27.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:26 smithi016 systemd[1]: Stopping Ceph mon.a for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:22:31.822 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a[31401]: 2024-04-13T03:22:31.554+0000 7f598d600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:22:31.822 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a[31401]: 2024-04-13T03:22:31.554+0000 7f598d600700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-13T03:22:32.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:31 smithi016 podman[31647]: 2024-04-13 03:22:31.822061127 +0000 UTC m=+4.528421357 container died 97e987feaaeab22c60101bd5b15e6c647c8b5414b771aed81c2483cc007ad19c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-13T03:22:36.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:36 smithi016 podman[31647]: 2024-04-13 03:22:36.322671485 +0000 UTC m=+9.029031748 container cleanup 97e987feaaeab22c60101bd5b15e6c647c8b5414b771aed81c2483cc007ad19c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-13T03:22:36.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:36 smithi016 bash[31647]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a 2024-04-13T03:22:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:36 smithi016 podman[31660]: 2024-04-13 03:22:36.606574182 +0000 UTC m=+4.764812180 container remove 97e987feaaeab22c60101bd5b15e6c647c8b5414b771aed81c2483cc007ad19c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-13T03:22:37.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:36 smithi016 systemd[1]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-13T03:22:37.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:36 smithi016 systemd[1]: Stopped Ceph mon.a for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:22:37.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:36 smithi016 systemd[1]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.a.service: Consumed 1.956s CPU time. 2024-04-13T03:22:37.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:36 smithi016 systemd[1]: Starting Ceph mon.a for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:22:37.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 podman[31753]: 2024-04-13 03:22:37.265765092 +0000 UTC m=+0.124355161 container create c421eb666bd809ce4d8d64dd3171b5653e89a67954ff15c15e7e2ec85a86fa4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-13T03:22:37.532 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 podman[31753]: 2024-04-13 03:22:37.177521807 +0000 UTC m=+0.036111858 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:22:37.627 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-13T03:22:37.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 podman[31753]: 2024-04-13 03:22:37.532029508 +0000 UTC m=+0.390619584 container init c421eb666bd809ce4d8d64dd3171b5653e89a67954ff15c15e7e2ec85a86fa4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-13T03:22:37.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 podman[31753]: 2024-04-13 03:22:37.543426206 +0000 UTC m=+0.402016281 container start c421eb666bd809ce4d8d64dd3171b5653e89a67954ff15c15e7e2ec85a86fa4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-13T03:22:37.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: set uid:gid to 167:167 (ceph:ceph) 2024-04-13T03:22:37.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev), process ceph-mon, pid 2 2024-04-13T03:22:37.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: pidfile_write: ignore empty --pid-file 2024-04-13T03:22:37.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: load: jerasure load: lrc 2024-04-13T03:22:37.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: RocksDB version: 7.9.2 2024-04-13T03:22:37.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Git sha 0 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Compile date 2024-04-12 04:47:35 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: DB SUMMARY 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: DB Session ID: UTA7830N3C62ZPXNPEVG 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: CURRENT file: CURRENT 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: IDENTITY file: IDENTITY 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105624 ; 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.error_if_exists: 0 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.create_if_missing: 0 2024-04-13T03:22:37.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.paranoid_checks: 1 2024-04-13T03:22:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-13T03:22:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-13T03:22:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-13T03:22:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.env: 0x55d68d772cc0 2024-04-13T03:22:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.fs: PosixFileSystem 2024-04-13T03:22:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.info_log: 0x55d6954f3c40 2024-04-13T03:22:37.858 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_file_opening_threads: 16 2024-04-13T03:22:37.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.statistics: (nil) 2024-04-13T03:22:37.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.use_fsync: 0 2024-04-13T03:22:37.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_log_file_size: 0 2024-04-13T03:22:37.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-13T03:22:37.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-13T03:22:37.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.keep_log_file_num: 1000 2024-04-13T03:22:37.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.recycle_log_file_num: 0 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.allow_fallocate: 1 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.allow_mmap_reads: 0 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.allow_mmap_writes: 0 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.use_direct_reads: 0 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.create_missing_column_families: 0 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.db_log_dir: 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.wal_dir: 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-13T03:22:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.advise_random_on_open: 1 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.db_write_buffer_size: 0 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.write_buffer_manager: 0x55d6954c41e0 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.rate_limiter: (nil) 2024-04-13T03:22:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-13T03:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.wal_recovery_mode: 2 2024-04-13T03:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.enable_thread_tracking: 0 2024-04-13T03:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.enable_pipelined_write: 0 2024-04-13T03:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.unordered_write: 0 2024-04-13T03:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-13T03:22:37.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.row_cache: None 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.wal_filter: None 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.allow_ingest_behind: 0 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.two_write_queues: 0 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.manual_wal_flush: 0 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.wal_compression: 0 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.atomic_flush: 0 2024-04-13T03:22:37.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.log_readahead_size: 0 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.best_efforts_recovery: 0 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.allow_data_in_errors: 0 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.db_host_id: __hostname__ 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-13T03:22:37.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_background_jobs: 2 2024-04-13T03:22:37.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_background_compactions: -1 2024-04-13T03:22:37.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_subcompactions: 1 2024-04-13T03:22:37.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-13T03:22:37.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-13T03:22:37.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-13T03:22:37.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_total_wal_size: 0 2024-04-13T03:22:37.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-13T03:22:37.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-13T03:22:37.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_open_files: -1 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bytes_per_sync: 0 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_readahead_size: 0 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_background_flushes: -1 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Compression algorithms supported: 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: kZSTD supported: 0 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: kXpressCompression supported: 0 2024-04-13T03:22:37.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: kLZ4HCCompression supported: 1 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: kZlibCompression supported: 1 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: kSnappyCompression supported: 1 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: kLZ4Compression supported: 1 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: kBZip2Compression supported: 0 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.merge_operator: 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_filter: None 2024-04-13T03:22:37.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_filter_factory: None 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.sst_partitioner_factory: None 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d6954f3e00) 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-04-13T03:22:37.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x55d6954f7090 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-04-13T03:22:37.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-04-13T03:22:37.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-04-13T03:22:37.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-04-13T03:22:37.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-04-13T03:22:37.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-04-13T03:22:37.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.write_buffer_size: 33554432 2024-04-13T03:22:37.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_write_buffer_number: 2 2024-04-13T03:22:37.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression: NoCompression 2024-04-13T03:22:37.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression: Disabled 2024-04-13T03:22:37.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.prefix_extractor: nullptr 2024-04-13T03:22:37.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-13T03:22:37.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.num_levels: 7 2024-04-13T03:22:37.873 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-13T03:22:37.873 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-13T03:22:37.873 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-13T03:22:37.873 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-13T03:22:37.873 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-13T03:22:37.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-13T03:22:37.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-13T03:22:37.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-13T03:22:37.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-13T03:22:37.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-13T03:22:37.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-13T03:22:37.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-13T03:22:37.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-13T03:22:37.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression_opts.level: 32767 2024-04-13T03:22:37.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression_opts.strategy: 0 2024-04-13T03:22:37.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-13T03:22:37.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression_opts.enabled: false 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.target_file_size_base: 67108864 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-13T03:22:37.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.arena_block_size: 1048576 2024-04-13T03:22:37.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.disable_auto_compactions: 0 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-13T03:22:37.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.inplace_update_support: 0 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.bloom_locality: 0 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.max_successive_merges: 0 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.paranoid_file_checks: 0 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.force_consistency_checks: 1 2024-04-13T03:22:37.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.report_bg_io_stats: 0 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.ttl: 2592000 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.enable_blob_files: false 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.min_blob_size: 0 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.blob_file_size: 268435456 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-13T03:22:37.880 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.blob_file_starting_level: 0 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f635cc0e-5b2a-4aec-b753-73600546dfb2 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712978557602570, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712978557604228, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100679, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98573, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14298, "raw_average_key_size": 49, "raw_value_size": 90841, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712978557, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f635cc0e-5b2a-4aec-b753-73600546dfb2", "db_session_id": "UTA7830N3C62ZPXNPEVG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712978557604405, "job": 1, "event": "recovery_finished"} 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d6955e6000 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: DB pointer 0x55d6955d0000 2024-04-13T03:22:37.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-13T03:22:37.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-13T03:22:37.883 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-13T03:22:37.883 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 100.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-13T03:22:37.883 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 100.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-13T03:22:37.883 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-13T03:22:37.883 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-13T03:22:37.883 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-13T03:22:37.883 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-13T03:22:37.883 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-13T03:22:37.883 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 92.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 15.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 15.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-13T03:22:37.884 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x55d6954f7090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mon.a@-1(???) e1 preinit fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mon.a@-1(???).mds e1 new map 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mon.a@-1(???).mds e1 print_map 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-13T03:22:37.885 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 bash[31753]: c421eb666bd809ce4d8d64dd3171b5653e89a67954ff15c15e7e2ec85a86fa4a 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 systemd[1]: Started Ceph mon.a for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-13T03:22:37.886 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: monmap epoch 1 2024-04-13T03:22:37.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:37.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: last_changed 2024-04-13T03:22:18.286080+0000 2024-04-13T03:22:37.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:22:37.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: min_mon_release 19 (squid) 2024-04-13T03:22:37.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: election_strategy: 1 2024-04-13T03:22:37.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:22:37.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: fsmap 2024-04-13T03:22:37.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: osdmap e1: 0 total, 0 up, 0 in 2024-04-13T03:22:37.887 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:37 smithi016 ceph-mon[31772]: mgrmap e1: no daemons active 2024-04-13T03:22:38.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.249+0000 7fed19600700 1 Processor -- start 2024-04-13T03:22:38.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.249+0000 7fed19600700 1 -- start start 2024-04-13T03:22:38.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.249+0000 7fed19600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed14108ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:38.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.250+0000 7fed19600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed14109090 con 0x7fed141086a0 2024-04-13T03:22:38.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.250+0000 7fed12a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed14108ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:38.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.250+0000 7fed12a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed14108ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32964/0 (socket says 172.21.15.16:32964) 2024-04-13T03:22:38.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.250+0000 7fed12a00700 1 -- 172.21.15.16:0/1360467816 learned_addr learned my addr 172.21.15.16:0/1360467816 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:38.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.251+0000 7fed12a00700 1 -- 172.21.15.16:0/1360467816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed14109800 con 0x7fed141086a0 2024-04-13T03:22:38.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.251+0000 7fed12a00700 1 --2- 172.21.15.16:0/1360467816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed14108ac0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed040098a0 tx=0x7fed04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3582d4736c4731 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:38.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.251+0000 7fed11600700 1 -- 172.21.15.16:0/1360467816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed04004160 con 0x7fed141086a0 2024-04-13T03:22:38.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.252+0000 7fed11600700 1 -- 172.21.15.16:0/1360467816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fed04004300 con 0x7fed141086a0 2024-04-13T03:22:38.934 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.252+0000 7fed11600700 1 -- 172.21.15.16:0/1360467816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed04011b50 con 0x7fed141086a0 2024-04-13T03:22:38.934 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.252+0000 7fed19600700 1 -- 172.21.15.16:0/1360467816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 msgr2=0x7fed14108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:38.934 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.252+0000 7fed19600700 1 --2- 172.21.15.16:0/1360467816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed14108ac0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed040098a0 tx=0x7fed04009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:38.934 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.252+0000 7fed19600700 1 -- 172.21.15.16:0/1360467816 shutdown_connections 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.252+0000 7fed19600700 1 --2- 172.21.15.16:0/1360467816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed14108ac0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.252+0000 7fed19600700 1 -- 172.21.15.16:0/1360467816 >> 172.21.15.16:0/1360467816 conn(0x7fed14103ce0 msgr2=0x7fed14106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.253+0000 7fed19600700 1 -- 172.21.15.16:0/1360467816 shutdown_connections 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.253+0000 7fed19600700 1 -- 172.21.15.16:0/1360467816 wait complete. 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.254+0000 7fed19600700 1 Processor -- start 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.254+0000 7fed19600700 1 -- start start 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.254+0000 7fed19600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed1407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.254+0000 7fed19600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fed1410cbb0 con 0x7fed141086a0 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.254+0000 7fed12a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed1407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.254+0000 7fed12a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed1407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32976/0 (socket says 172.21.15.16:32976) 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.254+0000 7fed12a00700 1 -- 172.21.15.16:0/2131406016 learned_addr learned my addr 172.21.15.16:0/2131406016 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.255+0000 7fed12a00700 1 -- 172.21.15.16:0/2131406016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1407e0a0 con 0x7fed141086a0 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.255+0000 7fed12a00700 1 --2- 172.21.15.16:0/2131406016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed1407db60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed04000c00 tx=0x7fed04011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:38.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.256+0000 7fed0b400700 1 -- 172.21.15.16:0/2131406016 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed04003b20 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.256+0000 7fed0b400700 1 -- 172.21.15.16:0/2131406016 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fed04003cc0 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.256+0000 7fed19600700 1 -- 172.21.15.16:0/2131406016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1407c290 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.256+0000 7fed0b400700 1 -- 172.21.15.16:0/2131406016 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed04024910 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.256+0000 7fed19600700 1 -- 172.21.15.16:0/2131406016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1407c7d0 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.256+0000 7fed0b400700 1 -- 172.21.15.16:0/2131406016 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fed04022020 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.256+0000 7fed0b400700 1 -- 172.21.15.16:0/2131406016 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed04011e00 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.257+0000 7fed19600700 1 -- 172.21.15.16:0/2131406016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed14064110 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.260+0000 7fed0b400700 1 -- 172.21.15.16:0/2131406016 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fed04042e50 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.338+0000 7fed19600700 1 -- 172.21.15.16:0/2131406016 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fed1407cee0 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.340+0000 7fed0b400700 1 -- 172.21.15.16:0/2131406016 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fed04031020 con 0x7fed141086a0 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.342+0000 7fed19600700 1 -- 172.21.15.16:0/2131406016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 msgr2=0x7fed1407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:38.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.342+0000 7fed19600700 1 --2- 172.21.15.16:0/2131406016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed1407db60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed04000c00 tx=0x7fed04011c10 comp rx=0 tx=0).stop 2024-04-13T03:22:38.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.342+0000 7fed19600700 1 -- 172.21.15.16:0/2131406016 shutdown_connections 2024-04-13T03:22:38.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.342+0000 7fed19600700 1 --2- 172.21.15.16:0/2131406016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed141086a0 0x7fed1407db60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:38.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.342+0000 7fed19600700 1 -- 172.21.15.16:0/2131406016 >> 172.21.15.16:0/2131406016 conn(0x7fed14103ce0 msgr2=0x7fed14104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:38.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.342+0000 7fed19600700 1 -- 172.21.15.16:0/2131406016 shutdown_connections 2024-04-13T03:22:38.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:38.342+0000 7fed19600700 1 -- 172.21.15.16:0/2131406016 wait complete. 2024-04-13T03:22:38.937 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-13T03:22:38.957 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-13T03:22:38.957 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-04-13T03:22:38.959 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-13T03:22:38.960 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-13T03:22:39.349 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.x 2024-04-13T03:22:39.349 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.x.service: Unit ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.x.service not loaded. 2024-04-13T03:22:39.596 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968.target.wants/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@.service. 2024-04-13T03:22:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:39 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2131406016' entity='client.admin' 2024-04-13T03:22:40.017 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:39 smithi016 systemd[1]: Starting Ceph mgr.x for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:22:40.289 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:40 smithi016 podman[31994]: 2024-04-13 03:22:39.929192602 +0000 UTC m=+0.044133285 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:22:40.289 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:40 smithi016 podman[31994]: 2024-04-13 03:22:40.034446255 +0000 UTC m=+0.149386882 container create d7830156b6df1fb06254e00795f1f6eb7e549a96785ebe2f0d33c501db2d9d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-13T03:22:40.289 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:40 smithi016 podman[31994]: 2024-04-13 03:22:40.288066808 +0000 UTC m=+0.403007508 container init d7830156b6df1fb06254e00795f1f6eb7e549a96785ebe2f0d33c501db2d9d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-13T03:22:40.418 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-13T03:22:40.418 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-13T03:22:40.438 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-13T03:22:40.439 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-13T03:22:40.439 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-13T03:22:40.439 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-04-13T03:22:40.462 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-13T03:22:40.462 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-13T03:22:40.478 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-13T03:22:40.478 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-13T03:22:40.478 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-13T03:22:40.478 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-13T03:22:40.478 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-04-13T03:22:40.478 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-04-13T03:22:40.605 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:40 smithi016 podman[31994]: 2024-04-13 03:22:40.298632432 +0000 UTC m=+0.413573062 container start d7830156b6df1fb06254e00795f1f6eb7e549a96785ebe2f0d33c501db2d9d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-13T03:22:40.605 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:40 smithi016 bash[31994]: d7830156b6df1fb06254e00795f1f6eb7e549a96785ebe2f0d33c501db2d9d3b 2024-04-13T03:22:40.605 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:40 smithi016 systemd[1]: Started Ceph mgr.x for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:22:41.158 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:41.115+0000 7f07dcb97200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T03:22:41.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:41 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3887914163' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-13T03:22:41.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:41.251+0000 7f07dcb97200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T03:22:42.278 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:42 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:42.095+0000 7f07dcb97200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T03:22:42.278 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:42 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:42.222+0000 7f07dcb97200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T03:22:42.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-13T03:22:42.298 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-13T03:22:42.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-13T03:22:42.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:42.301 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-13T03:22:22.658969+0000", 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.062+0000 7f9de8a00700 1 Processor -- start 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.062+0000 7f9de8a00700 1 -- start start 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.062+0000 7f9de8a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de4108910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.062+0000 7f9de8a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9de4108ee0 con 0x7f9de41084f0 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.063+0000 7f9de2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de4108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.063+0000 7f9de2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de4108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56650/0 (socket says 172.21.15.16:56650) 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.063+0000 7f9de2800700 1 -- 172.21.15.16:0/55379580 learned_addr learned my addr 172.21.15.16:0/55379580 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.064+0000 7f9de2800700 1 -- 172.21.15.16:0/55379580 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de4109650 con 0x7f9de41084f0 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.064+0000 7f9de2800700 1 --2- 172.21.15.16:0/55379580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de4108910 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40098a0 tx=0x7f9dd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621ae56e1f2f48ac server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.064+0000 7f9de1400700 1 -- 172.21.15.16:0/55379580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd4004160 con 0x7f9de41084f0 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.065+0000 7f9de1400700 1 -- 172.21.15.16:0/55379580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9dd4004300 con 0x7f9de41084f0 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.065+0000 7f9de1400700 1 -- 172.21.15.16:0/55379580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd4011b50 con 0x7f9de41084f0 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.065+0000 7f9de8a00700 1 -- 172.21.15.16:0/55379580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 msgr2=0x7f9de4108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.065+0000 7f9de8a00700 1 --2- 172.21.15.16:0/55379580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de4108910 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40098a0 tx=0x7f9dd4009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.065+0000 7f9de8a00700 1 -- 172.21.15.16:0/55379580 shutdown_connections 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.065+0000 7f9de8a00700 1 --2- 172.21.15.16:0/55379580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de4108910 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.065+0000 7f9de8a00700 1 -- 172.21.15.16:0/55379580 >> 172.21.15.16:0/55379580 conn(0x7f9de4103b70 msgr2=0x7f9de4105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.066+0000 7f9de8a00700 1 -- 172.21.15.16:0/55379580 shutdown_connections 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.066+0000 7f9de8a00700 1 -- 172.21.15.16:0/55379580 wait complete. 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.067+0000 7f9de8a00700 1 Processor -- start 2024-04-13T03:22:42.303 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.067+0000 7f9de8a00700 1 -- start start 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.067+0000 7f9de8a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.067+0000 7f9de8a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9de410ca00 con 0x7f9de41084f0 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.068+0000 7f9de2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.068+0000 7f9de2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56656/0 (socket says 172.21.15.16:56656) 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.068+0000 7f9de2800700 1 -- 172.21.15.16:0/3887914163 learned_addr learned my addr 172.21.15.16:0/3887914163 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.068+0000 7f9de2800700 1 -- 172.21.15.16:0/3887914163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de407e0a0 con 0x7f9de41084f0 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.068+0000 7f9de2800700 1 --2- 172.21.15.16:0/3887914163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de407db60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4000c00 tx=0x7f9dd4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.069+0000 7f9ddb400700 1 -- 172.21.15.16:0/3887914163 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd4003b20 con 0x7f9de41084f0 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.069+0000 7f9ddb400700 1 -- 172.21.15.16:0/3887914163 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9dd4003cc0 con 0x7f9de41084f0 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.069+0000 7f9ddb400700 1 -- 172.21.15.16:0/3887914163 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd4024910 con 0x7f9de41084f0 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.069+0000 7f9de8a00700 1 -- 172.21.15.16:0/3887914163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de407c290 con 0x7f9de41084f0 2024-04-13T03:22:42.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.069+0000 7f9de8a00700 1 -- 172.21.15.16:0/3887914163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de407c7d0 con 0x7f9de41084f0 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.069+0000 7f9ddb400700 1 -- 172.21.15.16:0/3887914163 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9dd4022020 con 0x7f9de41084f0 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.069+0000 7f9ddb400700 1 -- 172.21.15.16:0/3887914163 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9dd4011e00 con 0x7f9de41084f0 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.070+0000 7f9de8a00700 1 -- 172.21.15.16:0/3887914163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9de4064110 con 0x7f9de41084f0 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.074+0000 7f9ddb400700 1 -- 172.21.15.16:0/3887914163 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9dd401acd0 con 0x7f9de41084f0 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.156+0000 7f9de8a00700 1 -- 172.21.15.16:0/3887914163 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9de407ce70 con 0x7f9de41084f0 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.156+0000 7f9ddb400700 1 -- 172.21.15.16:0/3887914163 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9dd4032030 con 0x7f9de41084f0 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.157+0000 7f9de8a00700 1 -- 172.21.15.16:0/3887914163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 msgr2=0x7f9de407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.157+0000 7f9de8a00700 1 --2- 172.21.15.16:0/3887914163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de407db60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4000c00 tx=0x7f9dd4011c10 comp rx=0 tx=0).stop 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.158+0000 7f9de8a00700 1 -- 172.21.15.16:0/3887914163 shutdown_connections 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.158+0000 7f9de8a00700 1 --2- 172.21.15.16:0/3887914163 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de41084f0 0x7f9de407db60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.158+0000 7f9de8a00700 1 -- 172.21.15.16:0/3887914163 >> 172.21.15.16:0/3887914163 conn(0x7f9de4103b70 msgr2=0x7f9de41046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.158+0000 7f9de8a00700 1 -- 172.21.15.16:0/3887914163 shutdown_connections 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:41.158+0000 7f9de8a00700 1 -- 172.21.15.16:0/3887914163 wait complete. 2024-04-13T03:22:42.305 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-04-13T03:22:42.537 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:42 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:42.407+0000 7f07dcb97200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T03:22:42.537 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:42 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:42.536+0000 7f07dcb97200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T03:22:42.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:42 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:42.650+0000 7f07dcb97200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T03:22:43.260 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:42 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:42.997+0000 7f07dcb97200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T03:22:43.260 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:43 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:43.142+0000 7f07dcb97200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T03:22:43.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:43 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:43.259+0000 7f07dcb97200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T03:22:43.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:43 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:43.389+0000 7f07dcb97200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T03:22:44.104 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:43 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:43.612+0000 7f07dcb97200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T03:22:45.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:45 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/227507107' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-13T03:22:45.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:45.003+0000 7f07dcb97200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T03:22:45.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:45.207+0000 7f07dcb97200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T03:22:45.752 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:45.649+0000 7f07dcb97200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T03:22:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-13T03:22:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:22:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-13T03:22:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-13T03:22:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-13T03:22:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-13T03:22:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:45.763 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-13T03:22:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:45.765 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-13T03:22:45.765 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-13T03:22:45.772 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-13T03:22:45.773 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-13T03:22:45.774 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-13T03:22:22.658969+0000", 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.897+0000 7fd77b200700 1 Processor -- start 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.898+0000 7fd77b200700 1 -- start start 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.898+0000 7fd77b200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c107790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.898+0000 7fd77b200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c107d60 con 0x7fd77c107370 2024-04-13T03:22:45.775 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.898+0000 7fd779e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c107790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.899+0000 7fd779e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c107790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56660/0 (socket says 172.21.15.16:56660) 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.899+0000 7fd779e00700 1 -- 172.21.15.16:0/2348378948 learned_addr learned my addr 172.21.15.16:0/2348378948 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.899+0000 7fd779e00700 1 -- 172.21.15.16:0/2348378948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c108570 con 0x7fd77c107370 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.899+0000 7fd779e00700 1 --2- 172.21.15.16:0/2348378948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c107790 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0098a0 tx=0x7fd76c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1ae1ae1741e566 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.900+0000 7fd778a00700 1 -- 172.21.15.16:0/2348378948 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd76c004160 con 0x7fd77c107370 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.900+0000 7fd778a00700 1 -- 172.21.15.16:0/2348378948 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd76c004300 con 0x7fd77c107370 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.900+0000 7fd778a00700 1 -- 172.21.15.16:0/2348378948 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd76c011b50 con 0x7fd77c107370 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.900+0000 7fd77b200700 1 -- 172.21.15.16:0/2348378948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 msgr2=0x7fd77c107790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.900+0000 7fd77b200700 1 --2- 172.21.15.16:0/2348378948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c107790 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0098a0 tx=0x7fd76c009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.900+0000 7fd77b200700 1 -- 172.21.15.16:0/2348378948 shutdown_connections 2024-04-13T03:22:45.776 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.900+0000 7fd77b200700 1 --2- 172.21.15.16:0/2348378948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c107790 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.900+0000 7fd77b200700 1 -- 172.21.15.16:0/2348378948 >> 172.21.15.16:0/2348378948 conn(0x7fd77c1029b0 msgr2=0x7fd77c104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.901+0000 7fd77b200700 1 -- 172.21.15.16:0/2348378948 shutdown_connections 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.901+0000 7fd77b200700 1 -- 172.21.15.16:0/2348378948 wait complete. 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.902+0000 7fd77b200700 1 Processor -- start 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.902+0000 7fd77b200700 1 -- start start 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.902+0000 7fd77b200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c192520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.902+0000 7fd77b200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c10b920 con 0x7fd77c107370 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.903+0000 7fd779e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c192520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.903+0000 7fd779e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c192520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56674/0 (socket says 172.21.15.16:56674) 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.903+0000 7fd779e00700 1 -- 172.21.15.16:0/227507107 learned_addr learned my addr 172.21.15.16:0/227507107 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.903+0000 7fd779e00700 1 -- 172.21.15.16:0/227507107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c192a60 con 0x7fd77c107370 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.903+0000 7fd779e00700 1 --2- 172.21.15.16:0/227507107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c192520 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd76c000c00 tx=0x7fd76c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:45.777 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.904+0000 7fd772a00700 1 -- 172.21.15.16:0/227507107 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd76c003b20 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.904+0000 7fd772a00700 1 -- 172.21.15.16:0/227507107 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd76c003cc0 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.904+0000 7fd77b200700 1 -- 172.21.15.16:0/227507107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77c190c50 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.904+0000 7fd772a00700 1 -- 172.21.15.16:0/227507107 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd76c024910 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.904+0000 7fd77b200700 1 -- 172.21.15.16:0/227507107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c191190 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.904+0000 7fd772a00700 1 -- 172.21.15.16:0/227507107 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd76c022020 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.904+0000 7fd772a00700 1 -- 172.21.15.16:0/227507107 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd76c011e00 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.905+0000 7fd77b200700 1 -- 172.21.15.16:0/227507107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77c064110 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.908+0000 7fd772a00700 1 -- 172.21.15.16:0/227507107 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd76c01acd0 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.989+0000 7fd77b200700 1 -- 172.21.15.16:0/227507107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd77c1961b0 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.990+0000 7fd772a00700 1 -- 172.21.15.16:0/227507107 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd76c032030 con 0x7fd77c107370 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.991+0000 7fd77b200700 1 -- 172.21.15.16:0/227507107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 msgr2=0x7fd77c192520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.991+0000 7fd77b200700 1 --2- 172.21.15.16:0/227507107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c192520 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd76c000c00 tx=0x7fd76c011c10 comp rx=0 tx=0).stop 2024-04-13T03:22:45.778 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.991+0000 7fd77b200700 1 -- 172.21.15.16:0/227507107 shutdown_connections 2024-04-13T03:22:45.779 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.991+0000 7fd77b200700 1 --2- 172.21.15.16:0/227507107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd77c107370 0x7fd77c192520 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:45.779 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.991+0000 7fd77b200700 1 -- 172.21.15.16:0/227507107 >> 172.21.15.16:0/227507107 conn(0x7fd77c1029b0 msgr2=0x7fd77c1038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:45.779 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.992+0000 7fd77b200700 1 -- 172.21.15.16:0/227507107 shutdown_connections 2024-04-13T03:22:45.779 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:44.993+0000 7fd77b200700 1 -- 172.21.15.16:0/227507107 wait complete. 2024-04-13T03:22:45.779 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-04-13T03:22:46.046 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:45.779+0000 7f07dcb97200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T03:22:46.301 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:46 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:46.046+0000 7f07dcb97200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:22:46.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:46 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:46.301+0000 7f07dcb97200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T03:22:47.045 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:46 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:46.694+0000 7f07dcb97200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:22:47.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:47 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:47.043+0000 7f07dcb97200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T03:22:47.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:47 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:47.358+0000 7f07dcb97200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T03:22:48.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:48 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2669206253' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-13T03:22:48.706 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:48 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:48.458+0000 7f07dcb97200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T03:22:49.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:49.023 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-13T03:22:49.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:22:49.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-13T03:22:49.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-13T03:22:49.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-13T03:22:49.024 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-13T03:22:49.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:49.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-13T03:22:49.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-13T03:22:49.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-13T03:22:49.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:49.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-13T03:22:49.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-13T03:22:49.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:49.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-13T03:22:49.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-13T03:22:49.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:49.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-13T03:22:49.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-13T03:22:49.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:49.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-13T03:22:49.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:49.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-13T03:22:49.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-13T03:22:49.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-13T03:22:49.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-13T03:22:49.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:49.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-13T03:22:49.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:49.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-13T03:22:49.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-13T03:22:49.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:49.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-13T03:22:49.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-13T03:22:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-13T03:22:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-13T03:22:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-13T03:22:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-13T03:22:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-13T03:22:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-13T03:22:49.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-13T03:22:22.658969+0000", 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.351+0000 7fb1e0e00700 1 Processor -- start 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.351+0000 7fb1e0e00700 1 -- start start 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.352+0000 7fb1e0e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc108910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.352+0000 7fb1e0e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc108e50 con 0x7fb1dc1084f0 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.352+0000 7fb1db400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:49.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.352+0000 7fb1db400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56686/0 (socket says 172.21.15.16:56686) 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.352+0000 7fb1db400700 1 -- 172.21.15.16:0/349835577 learned_addr learned my addr 172.21.15.16:0/349835577 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.352+0000 7fb1db400700 1 -- 172.21.15.16:0/349835577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc108fd0 con 0x7fb1dc1084f0 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.353+0000 7fb1db400700 1 --2- 172.21.15.16:0/349835577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc108910 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0098a0 tx=0x7fb1cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f1d025e36a527d server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.353+0000 7fb1daa00700 1 -- 172.21.15.16:0/349835577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1cc004160 con 0x7fb1dc1084f0 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.353+0000 7fb1daa00700 1 -- 172.21.15.16:0/349835577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb1cc004300 con 0x7fb1dc1084f0 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.353+0000 7fb1daa00700 1 -- 172.21.15.16:0/349835577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1cc011b50 con 0x7fb1dc1084f0 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.354+0000 7fb1e0e00700 1 -- 172.21.15.16:0/349835577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 msgr2=0x7fb1dc108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.354+0000 7fb1e0e00700 1 --2- 172.21.15.16:0/349835577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc108910 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0098a0 tx=0x7fb1cc009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.354+0000 7fb1e0e00700 1 -- 172.21.15.16:0/349835577 shutdown_connections 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.354+0000 7fb1e0e00700 1 --2- 172.21.15.16:0/349835577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc108910 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.354+0000 7fb1e0e00700 1 -- 172.21.15.16:0/349835577 >> 172.21.15.16:0/349835577 conn(0x7fb1dc103b70 msgr2=0x7fb1dc105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.354+0000 7fb1e0e00700 1 -- 172.21.15.16:0/349835577 shutdown_connections 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.354+0000 7fb1e0e00700 1 -- 172.21.15.16:0/349835577 wait complete. 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.355+0000 7fb1e0e00700 1 Processor -- start 2024-04-13T03:22:49.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.355+0000 7fb1e0e00700 1 -- start start 2024-04-13T03:22:49.040 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.355+0000 7fb1e0e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:49.040 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.356+0000 7fb1e0e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc10a4e0 con 0x7fb1dc1084f0 2024-04-13T03:22:49.040 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.356+0000 7fb1db400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:49.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.356+0000 7fb1db400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56700/0 (socket says 172.21.15.16:56700) 2024-04-13T03:22:49.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.356+0000 7fb1db400700 1 -- 172.21.15.16:0/2669206253 learned_addr learned my addr 172.21.15.16:0/2669206253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:49.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.356+0000 7fb1db400700 1 -- 172.21.15.16:0/2669206253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc19c950 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.356+0000 7fb1db400700 1 --2- 172.21.15.16:0/2669206253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc19c410 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc011b60 tx=0x7fb1cc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.357+0000 7fb1d8c00700 1 -- 172.21.15.16:0/2669206253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1cc003b20 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.357+0000 7fb1d8c00700 1 -- 172.21.15.16:0/2669206253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb1cc003cc0 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.357+0000 7fb1d8c00700 1 -- 172.21.15.16:0/2669206253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1cc024840 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.357+0000 7fb1e0e00700 1 -- 172.21.15.16:0/2669206253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1dc19cbe0 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.357+0000 7fb1e0e00700 1 -- 172.21.15.16:0/2669206253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1dc19f8d0 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.357+0000 7fb1d8c00700 1 -- 172.21.15.16:0/2669206253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb1cc022020 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.357+0000 7fb1d8c00700 1 -- 172.21.15.16:0/2669206253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb1cc011e00 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.358+0000 7fb1e0e00700 1 -- 172.21.15.16:0/2669206253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1dc064110 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.361+0000 7fb1d8c00700 1 -- 172.21.15.16:0/2669206253 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb1cc043e50 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.441+0000 7fb1e0e00700 1 -- 172.21.15.16:0/2669206253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb1dc19ff00 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.442+0000 7fb1d8c00700 1 -- 172.21.15.16:0/2669206253 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fb1cc032020 con 0x7fb1dc1084f0 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.443+0000 7fb1e0e00700 1 -- 172.21.15.16:0/2669206253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 msgr2=0x7fb1dc19c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:49.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.443+0000 7fb1e0e00700 1 --2- 172.21.15.16:0/2669206253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc19c410 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc011b60 tx=0x7fb1cc011c40 comp rx=0 tx=0).stop 2024-04-13T03:22:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.444+0000 7fb1e0e00700 1 -- 172.21.15.16:0/2669206253 shutdown_connections 2024-04-13T03:22:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.444+0000 7fb1e0e00700 1 --2- 172.21.15.16:0/2669206253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1dc1084f0 0x7fb1dc19c410 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.444+0000 7fb1e0e00700 1 -- 172.21.15.16:0/2669206253 >> 172.21.15.16:0/2669206253 conn(0x7fb1dc103b70 msgr2=0x7fb1dc1046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.444+0000 7fb1e0e00700 1 -- 172.21.15.16:0/2669206253 shutdown_connections 2024-04-13T03:22:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:48.444+0000 7fb1e0e00700 1 -- 172.21.15.16:0/2669206253 wait complete. 2024-04-13T03:22:49.044 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (3/15)... 2024-04-13T03:22:49.327 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:49.193+0000 7f07dcb97200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T03:22:49.327 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:49.326+0000 7f07dcb97200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T03:22:50.104 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:49.621+0000 7f07dcb97200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: Activating manager daemon x 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: mgrmap e2: x(active, starting, since 0.0125834s) 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: Manager daemon x is now available 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-13T03:22:50.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:49 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-13T03:22:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:50 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' 2024-04-13T03:22:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:50 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' 2024-04-13T03:22:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:50 smithi016 ceph-mon[31772]: from='mgr.14100 172.21.15.16:0/2354123003' entity='mgr.x' 2024-04-13T03:22:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:50 smithi016 ceph-mon[31772]: mgrmap e3: x(active, since 1.02305s) 2024-04-13T03:22:52.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:52 smithi016 ceph-mon[31772]: mgrmap e4: x(active, since 2s) 2024-04-13T03:22:52.722 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:52 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/142777758' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-13T03:22:52.734 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:52.734 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-13T03:22:52.735 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:22:52.735 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-04-13T03:22:52.735 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-13T03:22:52.736 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-13T03:22:52.736 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-13T03:22:52.736 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:52.736 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-13T03:22:52.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-13T03:22:52.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-04-13T03:22:52.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:52.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-13T03:22:52.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-04-13T03:22:52.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:52.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-13T03:22:52.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-13T03:22:52.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:52.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-13T03:22:52.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-13T03:22:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-13T03:22:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-13T03:22:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-13T03:22:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-13T03:22:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-13T03:22:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-13T03:22:52.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-13T03:22:52.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-13T03:22:52.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-13T03:22:52.744 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-13T03:22:52.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-04-13T03:22:22.658969+0000", 2024-04-13T03:22:52.745 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-13T03:22:52.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-04-13T03:22:52.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-13T03:22:52.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-13T03:22:52.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.737+0000 7f6bf0a00700 1 Processor -- start 2024-04-13T03:22:52.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.737+0000 7f6bf0a00700 1 -- start start 2024-04-13T03:22:52.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.738+0000 7f6bf0a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec10f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:52.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.738+0000 7f6bf0a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec10f960 con 0x7f6bec10f000 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.738+0000 7f6be9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec10f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.738+0000 7f6be9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec10f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34268/0 (socket says 172.21.15.16:34268) 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.738+0000 7f6be9e00700 1 -- 172.21.15.16:0/3655028526 learned_addr learned my addr 172.21.15.16:0/3655028526 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.739+0000 7f6be9e00700 1 -- 172.21.15.16:0/3655028526 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec10fae0 con 0x7f6bec10f000 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.739+0000 7f6be9e00700 1 --2- 172.21.15.16:0/3655028526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec10f420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0098a0 tx=0x7f6bdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55cb224bbf6e2dc server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.739+0000 7f6be9400700 1 -- 172.21.15.16:0/3655028526 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bdc004160 con 0x7f6bec10f000 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.739+0000 7f6be9400700 1 -- 172.21.15.16:0/3655028526 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6bdc004300 con 0x7f6bec10f000 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.739+0000 7f6be9400700 1 -- 172.21.15.16:0/3655028526 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bdc011b50 con 0x7f6bec10f000 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.740+0000 7f6bf0a00700 1 -- 172.21.15.16:0/3655028526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 msgr2=0x7f6bec10f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.740+0000 7f6bf0a00700 1 --2- 172.21.15.16:0/3655028526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec10f420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0098a0 tx=0x7f6bdc009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.740+0000 7f6bf0a00700 1 -- 172.21.15.16:0/3655028526 shutdown_connections 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.740+0000 7f6bf0a00700 1 --2- 172.21.15.16:0/3655028526 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec10f420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:52.751 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.740+0000 7f6bf0a00700 1 -- 172.21.15.16:0/3655028526 >> 172.21.15.16:0/3655028526 conn(0x7f6bec07de00 msgr2=0x7f6bec080220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.740+0000 7f6bf0a00700 1 -- 172.21.15.16:0/3655028526 shutdown_connections 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.740+0000 7f6bf0a00700 1 -- 172.21.15.16:0/3655028526 wait complete. 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.741+0000 7f6bf0a00700 1 Processor -- start 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.742+0000 7f6bf0a00700 1 -- start start 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.742+0000 7f6bf0a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec1a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.742+0000 7f6bf0a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec113090 con 0x7f6bec10f000 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.742+0000 7f6be9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec1a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.742+0000 7f6be9e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec1a2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34278/0 (socket says 172.21.15.16:34278) 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.742+0000 7f6be9e00700 1 -- 172.21.15.16:0/142777758 learned_addr learned my addr 172.21.15.16:0/142777758 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.743+0000 7f6be9e00700 1 -- 172.21.15.16:0/142777758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec1a30b0 con 0x7f6bec10f000 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.743+0000 7f6be9e00700 1 --2- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec1a2b70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc011670 tx=0x7f6bdc011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.743+0000 7f6be2a00700 1 -- 172.21.15.16:0/142777758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bdc0039c0 con 0x7f6bec10f000 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.743+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec1a3340 con 0x7f6bec10f000 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.743+0000 7f6be2a00700 1 -- 172.21.15.16:0/142777758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6bdc011e50 con 0x7f6bec10f000 2024-04-13T03:22:52.752 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.743+0000 7f6be2a00700 1 -- 172.21.15.16:0/142777758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bdc01a5c0 con 0x7f6bec10f000 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.744+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec1a6030 con 0x7f6bec10f000 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.745+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bec113210 con 0x7f6bec10f000 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.745+0000 7f6be2a00700 1 -- 172.21.15.16:0/142777758 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f6bdc01a870 con 0x7f6bec10f000 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.746+0000 7f6be2a00700 1 --2- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f6bd0039560 0x7f6bd003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.746+0000 7f6be2a00700 1 -- 172.21.15.16:0/142777758 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6bdc04d690 con 0x7f6bec10f000 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.746+0000 7f6be3e00700 1 --2- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f6bd0039560 0x7f6bd003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.747+0000 7f6be3e00700 1 --2- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f6bd0039560 0x7f6bd003ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4006fd0 tx=0x7f6bd4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:51.753+0000 7f6be2a00700 1 -- 172.21.15.16:0/142777758 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bdc04d3f0 con 0x7f6bec10f000 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.073+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6bec064110 con 0x7f6bec10f000 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.074+0000 7f6be2a00700 1 -- 172.21.15.16:0/142777758 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6bdc003b60 con 0x7f6bec10f000 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.077+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f6bd0039560 msgr2=0x7f6bd003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.077+0000 7f6bf0a00700 1 --2- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f6bd0039560 0x7f6bd003ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4006fd0 tx=0x7f6bd4006e40 comp rx=0 tx=0).stop 2024-04-13T03:22:52.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.077+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 msgr2=0x7f6bec1a2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:52.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.077+0000 7f6bf0a00700 1 --2- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec1a2b70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc011670 tx=0x7f6bdc011b60 comp rx=0 tx=0).stop 2024-04-13T03:22:52.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.077+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 shutdown_connections 2024-04-13T03:22:52.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.077+0000 7f6bf0a00700 1 --2- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f6bd0039560 0x7f6bd003ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:52.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.077+0000 7f6bf0a00700 1 --2- 172.21.15.16:0/142777758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6bec10f000 0x7f6bec1a2b70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:52.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.077+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 >> 172.21.15.16:0/142777758 conn(0x7f6bec07de00 msgr2=0x7f6bec07e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:52.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.078+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 shutdown_connections 2024-04-13T03:22:52.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:52.078+0000 7f6bf0a00700 1 -- 172.21.15.16:0/142777758 wait complete. 2024-04-13T03:22:52.755 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-04-13T03:22:53.950 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:53 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3147046469' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-13T03:22:54.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:54.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-04-13T03:22:54.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:22:54.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-13T03:22:54.299 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-13T03:22:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-13T03:22:54.306 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-13T03:22:54.306 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.384+0000 7fbb8bc00700 1 Processor -- start 2024-04-13T03:22:54.306 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.384+0000 7fbb8bc00700 1 -- start start 2024-04-13T03:22:54.306 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.385+0000 7fbb8bc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8407c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:54.306 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.385+0000 7fbb8bc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8407c810 con 0x7fbb8407de20 2024-04-13T03:22:54.306 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.385+0000 7fbb89200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8407c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.385+0000 7fbb89200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8407c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34286/0 (socket says 172.21.15.16:34286) 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.385+0000 7fbb89200700 1 -- 172.21.15.16:0/1234112296 learned_addr learned my addr 172.21.15.16:0/1234112296 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.386+0000 7fbb89200700 1 -- 172.21.15.16:0/1234112296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8407c990 con 0x7fbb8407de20 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.386+0000 7fbb89200700 1 --2- 172.21.15.16:0/1234112296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8407c2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098a0 tx=0x7fbb78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6dd458fb691e674 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.386+0000 7fbb83400700 1 -- 172.21.15.16:0/1234112296 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb78004160 con 0x7fbb8407de20 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.386+0000 7fbb83400700 1 -- 172.21.15.16:0/1234112296 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb78004300 con 0x7fbb8407de20 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.386+0000 7fbb83400700 1 -- 172.21.15.16:0/1234112296 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb78011b50 con 0x7fbb8407de20 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.387+0000 7fbb8bc00700 1 -- 172.21.15.16:0/1234112296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 msgr2=0x7fbb8407c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.387+0000 7fbb8bc00700 1 --2- 172.21.15.16:0/1234112296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8407c2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098a0 tx=0x7fbb78009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.387+0000 7fbb8bc00700 1 -- 172.21.15.16:0/1234112296 shutdown_connections 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.387+0000 7fbb8bc00700 1 --2- 172.21.15.16:0/1234112296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8407c2d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.387+0000 7fbb8bc00700 1 -- 172.21.15.16:0/1234112296 >> 172.21.15.16:0/1234112296 conn(0x7fbb84103c50 msgr2=0x7fbb84106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.387+0000 7fbb8bc00700 1 -- 172.21.15.16:0/1234112296 shutdown_connections 2024-04-13T03:22:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.387+0000 7fbb8bc00700 1 -- 172.21.15.16:0/1234112296 wait complete. 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.388+0000 7fbb8bc00700 1 Processor -- start 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.389+0000 7fbb8bc00700 1 -- start start 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.389+0000 7fbb8bc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8419e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.389+0000 7fbb8bc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8410ec00 con 0x7fbb8407de20 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.389+0000 7fbb89200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8419e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.389+0000 7fbb89200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8419e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34302/0 (socket says 172.21.15.16:34302) 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.389+0000 7fbb89200700 1 -- 172.21.15.16:0/3147046469 learned_addr learned my addr 172.21.15.16:0/3147046469 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.390+0000 7fbb89200700 1 -- 172.21.15.16:0/3147046469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8419ed20 con 0x7fbb8407de20 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.390+0000 7fbb89200700 1 --2- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8419e7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbb78011b60 tx=0x7fbb78011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.390+0000 7fbb81600700 1 -- 172.21.15.16:0/3147046469 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb78003b20 con 0x7fbb8407de20 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.390+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8419efb0 con 0x7fbb8407de20 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.390+0000 7fbb81600700 1 -- 172.21.15.16:0/3147046469 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb78003cc0 con 0x7fbb8407de20 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.390+0000 7fbb81600700 1 -- 172.21.15.16:0/3147046469 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb78024840 con 0x7fbb8407de20 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.391+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb841a1ca0 con 0x7fbb8407de20 2024-04-13T03:22:54.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.392+0000 7fbb81600700 1 -- 172.21.15.16:0/3147046469 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fbb78022020 con 0x7fbb8407de20 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.392+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb84064110 con 0x7fbb8407de20 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.392+0000 7fbb81600700 1 --2- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fbb6c039560 0x7fbb6c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.392+0000 7fbb81600700 1 -- 172.21.15.16:0/3147046469 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbb78011e00 con 0x7fbb8407de20 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.393+0000 7fbb83e00700 1 --2- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fbb6c039560 0x7fbb6c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.393+0000 7fbb83e00700 1 --2- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fbb6c039560 0x7fbb6c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbb70006fd0 tx=0x7fbb70006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.398+0000 7fbb81600700 1 -- 172.21.15.16:0/3147046469 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb7804d8f0 con 0x7fbb8407de20 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.629+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fbb841a1e70 con 0x7fbb8407de20 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.630+0000 7fbb81600700 1 -- 172.21.15.16:0/3147046469 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fbb7801a950 con 0x7fbb8407de20 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.634+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fbb6c039560 msgr2=0x7fbb6c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.634+0000 7fbb8bc00700 1 --2- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fbb6c039560 0x7fbb6c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbb70006fd0 tx=0x7fbb70006e40 comp rx=0 tx=0).stop 2024-04-13T03:22:54.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.634+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 msgr2=0x7fbb8419e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:54.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.634+0000 7fbb8bc00700 1 --2- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8419e7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbb78011b60 tx=0x7fbb78011c40 comp rx=0 tx=0).stop 2024-04-13T03:22:54.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.634+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 shutdown_connections 2024-04-13T03:22:54.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.634+0000 7fbb8bc00700 1 --2- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fbb6c039560 0x7fbb6c03ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:54.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.634+0000 7fbb8bc00700 1 --2- 172.21.15.16:0/3147046469 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb8407de20 0x7fbb8419e7e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:54.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.634+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 >> 172.21.15.16:0/3147046469 conn(0x7fbb84103c50 msgr2=0x7fbb841047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:54.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.635+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 shutdown_connections 2024-04-13T03:22:54.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:53.635+0000 7fbb8bc00700 1 -- 172.21.15.16:0/3147046469 wait complete. 2024-04-13T03:22:54.310 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-04-13T03:22:55.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:55 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1295617097' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-13T03:22:55.605 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:55 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: ignoring --setuser ceph since I am not root 2024-04-13T03:22:55.605 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:55 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: ignoring --setgroup ceph since I am not root 2024-04-13T03:22:56.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.928+0000 7fdbd8800700 1 Processor -- start 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.928+0000 7fdbd8800700 1 -- start start 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.928+0000 7fdbd8800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd010acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.928+0000 7fdbd8800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd010b2a0 con 0x7fdbd010a8b0 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.929+0000 7fdbd5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd010acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.929+0000 7fdbd5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd010acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34304/0 (socket says 172.21.15.16:34304) 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.929+0000 7fdbd5e00700 1 -- 172.21.15.16:0/2166643364 learned_addr learned my addr 172.21.15.16:0/2166643364 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.930+0000 7fdbd5e00700 1 -- 172.21.15.16:0/2166643364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd010ba60 con 0x7fdbd010a8b0 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.930+0000 7fdbd5e00700 1 --2- 172.21.15.16:0/2166643364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd010acd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40098a0 tx=0x7fdbc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6874b800ebe766ae server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.930+0000 7fdbd4a00700 1 -- 172.21.15.16:0/2166643364 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbc4004160 con 0x7fdbd010a8b0 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.930+0000 7fdbd4a00700 1 -- 172.21.15.16:0/2166643364 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdbc4004300 con 0x7fdbd010a8b0 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.930+0000 7fdbd4a00700 1 -- 172.21.15.16:0/2166643364 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbc4011b50 con 0x7fdbd010a8b0 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.931+0000 7fdbd8800700 1 -- 172.21.15.16:0/2166643364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 msgr2=0x7fdbd010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.931+0000 7fdbd8800700 1 --2- 172.21.15.16:0/2166643364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd010acd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40098a0 tx=0x7fdbc4009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.931+0000 7fdbd8800700 1 -- 172.21.15.16:0/2166643364 shutdown_connections 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.931+0000 7fdbd8800700 1 --2- 172.21.15.16:0/2166643364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd010acd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:56.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.931+0000 7fdbd8800700 1 -- 172.21.15.16:0/2166643364 >> 172.21.15.16:0/2166643364 conn(0x7fdbd007de00 msgr2=0x7fdbd01084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.931+0000 7fdbd8800700 1 -- 172.21.15.16:0/2166643364 shutdown_connections 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.931+0000 7fdbd8800700 1 -- 172.21.15.16:0/2166643364 wait complete. 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.932+0000 7fdbd8800700 1 Processor -- start 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.933+0000 7fdbd8800700 1 -- start start 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.933+0000 7fdbd8800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.933+0000 7fdbd8800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd010ee10 con 0x7fdbd010a8b0 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.933+0000 7fdbd5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.933+0000 7fdbd5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd019e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34314/0 (socket says 172.21.15.16:34314) 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.933+0000 7fdbd5e00700 1 -- 172.21.15.16:0/1295617097 learned_addr learned my addr 172.21.15.16:0/1295617097 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.934+0000 7fdbd5e00700 1 -- 172.21.15.16:0/1295617097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd019ed90 con 0x7fdbd010a8b0 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.934+0000 7fdbd5e00700 1 --2- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd019e850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009fb0 tx=0x7fdbc4011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.934+0000 7fdbcaa00700 1 -- 172.21.15.16:0/1295617097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbc40038b0 con 0x7fdbd010a8b0 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.934+0000 7fdbcaa00700 1 -- 172.21.15.16:0/1295617097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdbc4011e50 con 0x7fdbd010a8b0 2024-04-13T03:22:56.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.934+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd019f020 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.934+0000 7fdbcaa00700 1 -- 172.21.15.16:0/1295617097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbc4019550 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.934+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd01a1d10 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.936+0000 7fdbcaa00700 1 -- 172.21.15.16:0/1295617097 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fdbc4003a50 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.936+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbd0064110 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.936+0000 7fdbcaa00700 1 --2- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fdbb8039560 0x7fdbb803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.936+0000 7fdbcaa00700 1 -- 172.21.15.16:0/1295617097 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdbc404d8f0 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.936+0000 7fdbd5400700 1 --2- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fdbb8039560 0x7fdbb803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.937+0000 7fdbd5400700 1 --2- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fdbb8039560 0x7fdbb803ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0006fd0 tx=0x7fdbc0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:54.942+0000 7fdbcaa00700 1 -- 172.21.15.16:0/1295617097 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdbc404d310 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.232+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fdbd01a1ee0 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.286+0000 7fdbcaa00700 1 -- 172.21.15.16:0/1295617097 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fdbc400fda0 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.291+0000 7fdbcaa00700 1 -- 172.21.15.16:0/1295617097 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fdbc4019800 con 0x7fdbd010a8b0 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.291+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fdbb8039560 msgr2=0x7fdbb803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.291+0000 7fdbd8800700 1 --2- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fdbb8039560 0x7fdbb803ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0006fd0 tx=0x7fdbc0006e40 comp rx=0 tx=0).stop 2024-04-13T03:22:56.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.291+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 msgr2=0x7fdbd019e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.291+0000 7fdbd8800700 1 --2- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd019e850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009fb0 tx=0x7fdbc4011bf0 comp rx=0 tx=0).stop 2024-04-13T03:22:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.291+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 shutdown_connections 2024-04-13T03:22:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.291+0000 7fdbd8800700 1 --2- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7fdbb8039560 0x7fdbb803ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.291+0000 7fdbd8800700 1 --2- 172.21.15.16:0/1295617097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbd010a8b0 0x7fdbd019e850 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009fb0 tx=0x7fdbc4011bf0 comp rx=0 tx=0).stop 2024-04-13T03:22:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.291+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 >> 172.21.15.16:0/1295617097 conn(0x7fdbd007de00 msgr2=0x7fdbd0107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.292+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 shutdown_connections 2024-04-13T03:22:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:55.292+0000 7fdbd8800700 1 -- 172.21.15.16:0/1295617097 wait complete. 2024-04-13T03:22:56.265 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:56 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:56.136+0000 7f364691e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T03:22:56.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:56 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1295617097' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-13T03:22:56.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:56 smithi016 ceph-mon[31772]: mgrmap e5: x(active, since 5s) 2024-04-13T03:22:56.571 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:56 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:56.261+0000 7f364691e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T03:22:57.213 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:57 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:57.101+0000 7f364691e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T03:22:57.546 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:22:57 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3037846299' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-13T03:22:57.547 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:57 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:57.232+0000 7f364691e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T03:22:57.547 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:57 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:57.417+0000 7f364691e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T03:22:57.547 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:57 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:57.545+0000 7f364691e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T03:22:57.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.569+0000 7f24f7200700 1 Processor -- start 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.569+0000 7f24f7200700 1 -- start start 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.570+0000 7f24f7200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f0107970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.570+0000 7f24f7200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24f0107f40 con 0x7f24f0107550 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.570+0000 7f24efe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f0107970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.570+0000 7f24efe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f0107970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34342/0 (socket says 172.21.15.16:34342) 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.570+0000 7f24efe00700 1 -- 172.21.15.16:0/1769060435 learned_addr learned my addr 172.21.15.16:0/1769060435 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.571+0000 7f24efe00700 1 -- 172.21.15.16:0/1769060435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f0108750 con 0x7f24f0107550 2024-04-13T03:22:57.579 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.571+0000 7f24efe00700 1 --2- 172.21.15.16:0/1769060435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f0107970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f24e00098a0 tx=0x7f24e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dece3c0570579f1f server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.571+0000 7f24eea00700 1 -- 172.21.15.16:0/1769060435 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e0004160 con 0x7f24f0107550 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.571+0000 7f24eea00700 1 -- 172.21.15.16:0/1769060435 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f24e0004300 con 0x7f24f0107550 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.571+0000 7f24eea00700 1 -- 172.21.15.16:0/1769060435 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e0011b50 con 0x7f24f0107550 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.572+0000 7f24f7200700 1 -- 172.21.15.16:0/1769060435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 msgr2=0x7f24f0107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.572+0000 7f24f7200700 1 --2- 172.21.15.16:0/1769060435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f0107970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f24e00098a0 tx=0x7f24e0009bb0 comp rx=0 tx=0).stop 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.572+0000 7f24f7200700 1 -- 172.21.15.16:0/1769060435 shutdown_connections 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.572+0000 7f24f7200700 1 --2- 172.21.15.16:0/1769060435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f0107970 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.572+0000 7f24f7200700 1 -- 172.21.15.16:0/1769060435 >> 172.21.15.16:0/1769060435 conn(0x7f24f0102b50 msgr2=0x7f24f0104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.572+0000 7f24f7200700 1 -- 172.21.15.16:0/1769060435 shutdown_connections 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.573+0000 7f24f7200700 1 -- 172.21.15.16:0/1769060435 wait complete. 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.573+0000 7f24f7200700 1 Processor -- start 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.574+0000 7f24f7200700 1 -- start start 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.574+0000 7f24f7200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f019e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:57.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.574+0000 7f24f7200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24f010bb00 con 0x7f24f0107550 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.574+0000 7f24efe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f019e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.574+0000 7f24efe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f019e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34356/0 (socket says 172.21.15.16:34356) 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.574+0000 7f24efe00700 1 -- 172.21.15.16:0/3037846299 learned_addr learned my addr 172.21.15.16:0/3037846299 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.575+0000 7f24efe00700 1 -- 172.21.15.16:0/3037846299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f019ee10 con 0x7f24f0107550 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.575+0000 7f24efe00700 1 --2- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f019e8d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f24e0011850 tx=0x7f24e0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.575+0000 7f24ecc00700 1 -- 172.21.15.16:0/3037846299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e0003af0 con 0x7f24f0107550 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.575+0000 7f24ecc00700 1 -- 172.21.15.16:0/3037846299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f24e0020070 con 0x7f24f0107550 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.575+0000 7f24ecc00700 1 -- 172.21.15.16:0/3037846299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e00195a0 con 0x7f24f0107550 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.575+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f019f0a0 con 0x7f24f0107550 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.575+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f01a1d90 con 0x7f24f0107550 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.576+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f0064110 con 0x7f24f0107550 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.578+0000 7f24ecc00700 1 -- 172.21.15.16:0/3037846299 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f24e001a430 con 0x7f24f0107550 2024-04-13T03:22:57.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.578+0000 7f24ecc00700 1 --2- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f24d4039600 0x7f24d403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.578+0000 7f24ecc00700 1 -- 172.21.15.16:0/3037846299 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f24e0011e00 con 0x7f24f0107550 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.578+0000 7f24ef400700 1 -- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f24d4039600 msgr2=0x7f24d403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2135228404 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.578+0000 7f24ef400700 1 --2- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f24d4039600 0x7f24d403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.582+0000 7f24ecc00700 1 -- 172.21.15.16:0/3037846299 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24e002a030 con 0x7f24f0107550 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.779+0000 7f24ef400700 1 -- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f24d4039600 msgr2=0x7f24d403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2135228404 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.779+0000 7f24ef400700 1 --2- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f24d4039600 0x7f24d403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.864+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f24f002e170 con 0x7f24f0107550 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.864+0000 7f24ecc00700 1 -- 172.21.15.16:0/3037846299 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f24e002de00 con 0x7f24f0107550 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.867+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f24d4039600 msgr2=0x7f24d403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.867+0000 7f24f7200700 1 --2- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f24d4039600 0x7f24d403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.867+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 msgr2=0x7f24f019e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.867+0000 7f24f7200700 1 --2- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f019e8d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f24e0011850 tx=0x7f24e0011c10 comp rx=0 tx=0).stop 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.867+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 shutdown_connections 2024-04-13T03:22:57.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.867+0000 7f24f7200700 1 --2- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f24d4039600 0x7f24d403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.867+0000 7f24f7200700 1 --2- 172.21.15.16:0/3037846299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f0107550 0x7f24f019e8d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:22:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.867+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 >> 172.21.15.16:0/3037846299 conn(0x7f24f0102b50 msgr2=0x7f24f0103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:22:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.867+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 shutdown_connections 2024-04-13T03:22:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:56.868+0000 7f24f7200700 1 -- 172.21.15.16:0/3037846299 wait complete. 2024-04-13T03:22:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-04-13T03:22:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-04-13T03:22:57.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:57 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:57.662+0000 7f364691e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T03:22:58.167 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:58 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:58.016+0000 7f364691e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T03:22:58.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:58 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:58.166+0000 7f364691e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T03:22:58.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:58 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:58.284+0000 7f364691e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T03:22:58.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:58 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:58.410+0000 7f364691e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T03:22:59.104 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:58 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:58.635+0000 7f364691e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T03:23:00.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:22:59 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:22:59.970+0000 7f364691e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T03:23:00.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:00.150+0000 7f364691e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T03:23:00.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:00.580+0000 7f364691e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T03:23:00.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:00.703+0000 7f364691e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T03:23:01.228 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:00.971+0000 7f364691e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:23:01.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:01 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:01.227+0000 7f364691e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T03:23:01.944 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:01 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:01.619+0000 7f364691e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:23:02.259 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:01 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:01.942+0000 7f364691e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T03:23:02.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:02 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:02.257+0000 7f364691e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T03:23:03.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:03 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:03.328+0000 7f364691e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T03:23:04.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:04.048+0000 7f364691e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T03:23:04.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:04.183+0000 7f364691e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T03:23:04.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:04.476+0000 7f364691e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T03:23:04.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: Active manager daemon x restarted 2024-04-13T03:23:04.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: Activating manager daemon x 2024-04-13T03:23:04.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: osdmap e2: 0 total, 0 up, 0 in 2024-04-13T03:23:04.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: mgrmap e6: x(active, starting, since 0.00957814s) 2024-04-13T03:23:04.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:23:04.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-13T03:23:04.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-13T03:23:04.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-13T03:23:04.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-13T03:23:04.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:04 smithi016 ceph-mon[31772]: Manager daemon x is now available 2024-04-13T03:23:05.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: Found migration_current of "None". Setting to last migration. 2024-04-13T03:23:05.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:05.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:05.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:05.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:05.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-13T03:23:05.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-13T03:23:05.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:05.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:05.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:05 smithi016 ceph-mon[31772]: mgrmap e7: x(active, since 1.01648s) 2024-04-13T03:23:06.206 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.180+0000 7f31d0000700 1 Processor -- start 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.180+0000 7f31d0000700 1 -- start start 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.181+0000 7f31d0000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c810b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.181+0000 7f31d0000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31c807bea0 con 0x7f31c81090c0 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.181+0000 7f31cd600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c810b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.181+0000 7f31cd600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c810b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34372/0 (socket says 172.21.15.16:34372) 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.181+0000 7f31cd600700 1 -- 172.21.15.16:0/497102047 learned_addr learned my addr 172.21.15.16:0/497102047 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.182+0000 7f31cd600700 1 -- 172.21.15.16:0/497102047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c810b9f0 con 0x7f31c81090c0 2024-04-13T03:23:06.207 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.182+0000 7f31cd600700 1 --2- 172.21.15.16:0/497102047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c810b4b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f31bc0098a0 tx=0x7f31bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b05870103b3325 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.182+0000 7f31c7e00700 1 -- 172.21.15.16:0/497102047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31bc004160 con 0x7f31c81090c0 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.183+0000 7f31c7e00700 1 -- 172.21.15.16:0/497102047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f31bc004300 con 0x7f31c81090c0 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.183+0000 7f31c7e00700 1 -- 172.21.15.16:0/497102047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31bc011b50 con 0x7f31c81090c0 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.183+0000 7f31d0000700 1 -- 172.21.15.16:0/497102047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 msgr2=0x7f31c810b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.183+0000 7f31d0000700 1 --2- 172.21.15.16:0/497102047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c810b4b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f31bc0098a0 tx=0x7f31bc009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.183+0000 7f31d0000700 1 -- 172.21.15.16:0/497102047 shutdown_connections 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.183+0000 7f31d0000700 1 --2- 172.21.15.16:0/497102047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c810b4b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.183+0000 7f31d0000700 1 -- 172.21.15.16:0/497102047 >> 172.21.15.16:0/497102047 conn(0x7f31c8102b50 msgr2=0x7f31c8104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.184+0000 7f31d0000700 1 -- 172.21.15.16:0/497102047 shutdown_connections 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.184+0000 7f31d0000700 1 -- 172.21.15.16:0/497102047 wait complete. 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.185+0000 7f31d0000700 1 Processor -- start 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.185+0000 7f31d0000700 1 -- start start 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.186+0000 7f31d0000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.186+0000 7f31d0000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f31c810ca20 con 0x7f31c81090c0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.186+0000 7f31cd600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.186+0000 7f31cd600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34380/0 (socket says 172.21.15.16:34380) 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.186+0000 7f31cd600700 1 -- 172.21.15.16:0/3455771360 learned_addr learned my addr 172.21.15.16:0/3455771360 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.187+0000 7f31cd600700 1 -- 172.21.15.16:0/3455771360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c819cbb0 con 0x7f31c81090c0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.187+0000 7f31cd600700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c819c670 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31bc011b60 tx=0x7f31bc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.187+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31bc003970 con 0x7f31c81090c0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.187+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f31bc003b10 con 0x7f31c81090c0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.187+0000 7f31d0000700 1 -- 172.21.15.16:0/3455771360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c819ce40 con 0x7f31c81090c0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.187+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31bc024610 con 0x7f31c81090c0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.188+0000 7f31d0000700 1 -- 172.21.15.16:0/3455771360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c819fb30 con 0x7f31c81090c0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.189+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f31bc01a9f0 con 0x7f31c81090c0 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.190+0000 7f31c6000700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 0x7f31b003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.190+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 --> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f31b003c1d0 con 0x7f31b0039600 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.190+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f31bc011e00 con 0x7f31c81090c0 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.190+0000 7f31ccc00700 1 -- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 msgr2=0x7f31b003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2135228404 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.190+0000 7f31ccc00700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 0x7f31b003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.391+0000 7f31ccc00700 1 -- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 msgr2=0x7f31b003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2135228404 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.391+0000 7f31ccc00700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 0x7f31b003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.792+0000 7f31ccc00700 1 -- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 msgr2=0x7f31b003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2135228404 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:58.792+0000 7f31ccc00700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 0x7f31b003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:59.593+0000 7f31ccc00700 1 -- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 msgr2=0x7f31b003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2135228404 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:22:59.593+0000 7f31ccc00700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 0x7f31b003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:01.194+0000 7f31ccc00700 1 -- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 msgr2=0x7f31b003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2135228404 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:01.195+0000 7f31ccc00700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 0x7f31b003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:04.398+0000 7f31ccc00700 1 -- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 msgr2=0x7f31b003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/2135228404 2024-04-13T03:23:06.210 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:04.398+0000 7f31ccc00700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 0x7f31b003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:04.486+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f31bc01a430 con 0x7f31c81090c0 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:04.487+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 msgr2=0x7f31b003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:04.487+0000 7f31c6000700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 0x7f31b003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.490+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f31bc04f330 con 0x7f31c81090c0 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.490+0000 7f31c6000700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f31b003cef0 0x7f31b003f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.490+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f31bc00af30 con 0x7f31b003cef0 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.492+0000 7f31ccc00700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f31b003cef0 0x7f31b003f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.494+0000 7f31ccc00700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f31b003cef0 0x7f31b003f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f31b4003a10 tx=0x7f31b40092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.495+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 <== mgr.14120 v2:172.21.15.16:6800/164620940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f31bc00af30 con 0x7f31b003cef0 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.507+0000 7f31d0000700 1 -- 172.21.15.16:0/3455771360 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f31c819d320 con 0x7f31b003cef0 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.508+0000 7f31c6000700 1 -- 172.21.15.16:0/3455771360 <== mgr.14120 v2:172.21.15.16:6800/164620940 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f31c819d320 con 0x7f31b003cef0 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 -- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f31b003cef0 msgr2=0x7f31b003f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f31b003cef0 0x7f31b003f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f31b4003a10 tx=0x7f31b40092b0 comp rx=0 tx=0).stop 2024-04-13T03:23:06.211 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 -- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 msgr2=0x7f31c819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c819c670 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31bc011b60 tx=0x7f31bc011c40 comp rx=0 tx=0).stop 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 -- 172.21.15.16:0/3455771360 shutdown_connections 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f31b003cef0 0x7f31b003f2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31c81090c0 0x7f31c819c670 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 --2- 172.21.15.16:0/3455771360 >> [v2:172.21.15.16:6800/2135228404,v1:172.21.15.16:6801/2135228404] conn(0x7f31b0039600 0x7f31b003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 -- 172.21.15.16:0/3455771360 >> 172.21.15.16:0/3455771360 conn(0x7f31c8102b50 msgr2=0x7f31c8103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.509+0000 7f31d0000700 1 -- 172.21.15.16:0/3455771360 shutdown_connections 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:05.510+0000 7f31d0000700 1 -- 172.21.15.16:0/3455771360 wait complete. 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-04-13T03:23:06.212 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-04-13T03:23:06.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:06 smithi016 ceph-mon[31772]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-13T03:23:06.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:06 smithi016 ceph-mon[31772]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-13T03:23:06.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:06 smithi016 ceph-mon[31772]: [13/Apr/2024:03:23:06] ENGINE Bus STARTING 2024-04-13T03:23:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.824+0000 7f7ef3200700 1 Processor -- start 2024-04-13T03:23:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.825+0000 7f7ef3200700 1 -- start start 2024-04-13T03:23:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.825+0000 7f7ef3200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef4108910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.825+0000 7f7ef3200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef4108e50 con 0x7f7ef41084f0 2024-04-13T03:23:07.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.825+0000 7f7ef1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef4108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.825+0000 7f7ef1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef4108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53776/0 (socket says 172.21.15.16:53776) 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.826+0000 7f7ef1e00700 1 -- 172.21.15.16:0/3782164612 learned_addr learned my addr 172.21.15.16:0/3782164612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.826+0000 7f7ef1e00700 1 -- 172.21.15.16:0/3782164612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef4108fd0 con 0x7f7ef41084f0 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.826+0000 7f7ef1e00700 1 --2- 172.21.15.16:0/3782164612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef4108910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40098a0 tx=0x7f7ee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90cbdece2df4319d server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.826+0000 7f7ef0a00700 1 -- 172.21.15.16:0/3782164612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ee4004160 con 0x7f7ef41084f0 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.826+0000 7f7ef0a00700 1 -- 172.21.15.16:0/3782164612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7ee4004300 con 0x7f7ef41084f0 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.826+0000 7f7ef0a00700 1 -- 172.21.15.16:0/3782164612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ee4011b50 con 0x7f7ef41084f0 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.827+0000 7f7ef3200700 1 -- 172.21.15.16:0/3782164612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 msgr2=0x7f7ef4108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.827+0000 7f7ef3200700 1 --2- 172.21.15.16:0/3782164612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef4108910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40098a0 tx=0x7f7ee4009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.827+0000 7f7ef3200700 1 -- 172.21.15.16:0/3782164612 shutdown_connections 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.827+0000 7f7ef3200700 1 --2- 172.21.15.16:0/3782164612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef4108910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.827+0000 7f7ef3200700 1 -- 172.21.15.16:0/3782164612 >> 172.21.15.16:0/3782164612 conn(0x7f7ef4103b70 msgr2=0x7f7ef4105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.827+0000 7f7ef3200700 1 -- 172.21.15.16:0/3782164612 shutdown_connections 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.827+0000 7f7ef3200700 1 -- 172.21.15.16:0/3782164612 wait complete. 2024-04-13T03:23:07.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.828+0000 7f7ef3200700 1 Processor -- start 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.829+0000 7f7ef3200700 1 -- start start 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.829+0000 7f7ef3200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.829+0000 7f7ef3200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef410a4e0 con 0x7f7ef41084f0 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.829+0000 7f7ef1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.829+0000 7f7ef1e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef419c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53786/0 (socket says 172.21.15.16:53786) 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.829+0000 7f7ef1e00700 1 -- 172.21.15.16:0/2100447030 learned_addr learned my addr 172.21.15.16:0/2100447030 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.829+0000 7f7ef1e00700 1 -- 172.21.15.16:0/2100447030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef419c950 con 0x7f7ef41084f0 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.830+0000 7f7ef1e00700 1 --2- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef419c410 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4011b60 tx=0x7f7ee4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.830+0000 7f7eeaa00700 1 -- 172.21.15.16:0/2100447030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ee4003b20 con 0x7f7ef41084f0 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.830+0000 7f7eeaa00700 1 -- 172.21.15.16:0/2100447030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7ee4003cc0 con 0x7f7ef41084f0 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.830+0000 7f7eeaa00700 1 -- 172.21.15.16:0/2100447030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ee4024840 con 0x7f7ef41084f0 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.830+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef419cbe0 con 0x7f7ef41084f0 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.830+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef419f8d0 con 0x7f7ef41084f0 2024-04-13T03:23:07.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.831+0000 7f7eeaa00700 1 -- 172.21.15.16:0/2100447030 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7ee4022020 con 0x7f7ef41084f0 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.831+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef4064110 con 0x7f7ef41084f0 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.832+0000 7f7eeaa00700 1 --2- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f7ed80395b0 0x7f7ed803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.832+0000 7f7eeaa00700 1 -- 172.21.15.16:0/2100447030 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7ee4011e00 con 0x7f7ef41084f0 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.832+0000 7f7ef1400700 1 --2- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f7ed80395b0 0x7f7ed803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.833+0000 7f7ef1400700 1 --2- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f7ed80395b0 0x7f7ed803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006fd0 tx=0x7f7ee0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:06.838+0000 7f7eeaa00700 1 -- 172.21.15.16:0/2100447030 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ee404d240 con 0x7f7ef41084f0 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.087+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f7ef4104db0 con 0x7f7ed80395b0 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.101+0000 7f7eeaa00700 1 -- 172.21.15.16:0/2100447030 <== mgr.14120 v2:172.21.15.16:6800/164620940 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7ef4104db0 con 0x7f7ed80395b0 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.104+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f7ed80395b0 msgr2=0x7f7ed803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.104+0000 7f7ef3200700 1 --2- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f7ed80395b0 0x7f7ed803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006fd0 tx=0x7f7ee0006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.104+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 msgr2=0x7f7ef419c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.104+0000 7f7ef3200700 1 --2- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef419c410 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4011b60 tx=0x7f7ee4011c40 comp rx=0 tx=0).stop 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.105+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 shutdown_connections 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.105+0000 7f7ef3200700 1 --2- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f7ed80395b0 0x7f7ed803ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:07.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.105+0000 7f7ef3200700 1 --2- 172.21.15.16:0/2100447030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ef41084f0 0x7f7ef419c410 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:07.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.105+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 >> 172.21.15.16:0/2100447030 conn(0x7f7ef4103b70 msgr2=0x7f7ef41046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:07.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.105+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 shutdown_connections 2024-04-13T03:23:07.714 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:07.105+0000 7f7ef3200700 1 -- 172.21.15.16:0/2100447030 wait complete. 2024-04-13T03:23:07.787 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:07 smithi016 ceph-mon[31772]: [13/Apr/2024:03:23:06] ENGINE Serving on https://172.21.15.16:7150 2024-04-13T03:23:07.787 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:07 smithi016 ceph-mon[31772]: mgrmap e8: x(active, since 2s) 2024-04-13T03:23:07.787 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:07 smithi016 ceph-mon[31772]: [13/Apr/2024:03:23:06] ENGINE Serving on http://172.21.15.16:8765 2024-04-13T03:23:07.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:07 smithi016 ceph-mon[31772]: [13/Apr/2024:03:23:06] ENGINE Bus STARTED 2024-04-13T03:23:07.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:07 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:07.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:07 smithi016 ceph-mon[31772]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:07.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:07 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:07.788 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:07 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:09.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-13T03:23:09.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.373+0000 7f412aa00700 1 Processor -- start 2024-04-13T03:23:09.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.373+0000 7f412aa00700 1 -- start start 2024-04-13T03:23:09.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.374+0000 7f412aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412410b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:09.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.374+0000 7f412aa00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f412407be20 con 0x7f4124109140 2024-04-13T03:23:09.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.374+0000 7f4123e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412410b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:09.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.374+0000 7f4123e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412410b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53790/0 (socket says 172.21.15.16:53790) 2024-04-13T03:23:09.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.374+0000 7f4123e00700 1 -- 172.21.15.16:0/4135310631 learned_addr learned my addr 172.21.15.16:0/4135310631 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:09.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.375+0000 7f4123e00700 1 -- 172.21.15.16:0/4135310631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412410ba70 con 0x7f4124109140 2024-04-13T03:23:09.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.375+0000 7f4123e00700 1 --2- 172.21.15.16:0/4135310631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412410b530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41140098a0 tx=0x7f4114009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6b66308c59bdb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:09.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.375+0000 7f4123400700 1 -- 172.21.15.16:0/4135310631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4114004160 con 0x7f4124109140 2024-04-13T03:23:09.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.375+0000 7f4123400700 1 -- 172.21.15.16:0/4135310631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4114004300 con 0x7f4124109140 2024-04-13T03:23:09.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.375+0000 7f4123400700 1 -- 172.21.15.16:0/4135310631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4114011b50 con 0x7f4124109140 2024-04-13T03:23:09.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.376+0000 7f412aa00700 1 -- 172.21.15.16:0/4135310631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 msgr2=0x7f412410b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:09.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.376+0000 7f412aa00700 1 --2- 172.21.15.16:0/4135310631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412410b530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41140098a0 tx=0x7f4114009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:09.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.377+0000 7f412aa00700 1 -- 172.21.15.16:0/4135310631 shutdown_connections 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.377+0000 7f412aa00700 1 --2- 172.21.15.16:0/4135310631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412410b530 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.377+0000 7f412aa00700 1 -- 172.21.15.16:0/4135310631 >> 172.21.15.16:0/4135310631 conn(0x7f4124102b50 msgr2=0x7f4124104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.377+0000 7f412aa00700 1 -- 172.21.15.16:0/4135310631 shutdown_connections 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.377+0000 7f412aa00700 1 -- 172.21.15.16:0/4135310631 wait complete. 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.378+0000 7f412aa00700 1 Processor -- start 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.378+0000 7f412aa00700 1 -- start start 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.379+0000 7f412aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.379+0000 7f412aa00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f412410ca30 con 0x7f4124109140 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.379+0000 7f4123e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.379+0000 7f4123e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412419c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53806/0 (socket says 172.21.15.16:53806) 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.379+0000 7f4123e00700 1 -- 172.21.15.16:0/418262200 learned_addr learned my addr 172.21.15.16:0/418262200 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.379+0000 7f4123e00700 1 -- 172.21.15.16:0/418262200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412419cbc0 con 0x7f4124109140 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.380+0000 7f4123e00700 1 --2- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412419c680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4114011b60 tx=0x7f4114011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.380+0000 7f4121600700 1 -- 172.21.15.16:0/418262200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4114003970 con 0x7f4124109140 2024-04-13T03:23:09.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.380+0000 7f4121600700 1 -- 172.21.15.16:0/418262200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4114003b10 con 0x7f4124109140 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.380+0000 7f4121600700 1 -- 172.21.15.16:0/418262200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4114024610 con 0x7f4124109140 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.380+0000 7f412aa00700 1 -- 172.21.15.16:0/418262200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412419ce50 con 0x7f4124109140 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.380+0000 7f412aa00700 1 -- 172.21.15.16:0/418262200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f412419fb40 con 0x7f4124109140 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.381+0000 7f4121600700 1 -- 172.21.15.16:0/418262200 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f411401a9f0 con 0x7f4124109140 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.381+0000 7f411a000700 1 -- 172.21.15.16:0/418262200 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40fc005330 con 0x7f4124109140 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.382+0000 7f4121600700 1 --2- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f4108039660 0x7f410803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.382+0000 7f4121600700 1 -- 172.21.15.16:0/418262200 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4114011e00 con 0x7f4124109140 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.382+0000 7f411be00700 1 --2- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f4108039660 0x7f410803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.383+0000 7f411be00700 1 --2- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f4108039660 0x7f410803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f410c006fd0 tx=0x7f410c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.388+0000 7f4121600700 1 -- 172.21.15.16:0/418262200 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4114024de0 con 0x7f4124109140 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.614+0000 7f411a000700 1 -- 172.21.15.16:0/418262200 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f40fc000bc0 con 0x7f4108039660 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.616+0000 7f4121600700 1 -- 172.21.15.16:0/418262200 <== mgr.14120 v2:172.21.15.16:6800/164620940 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f40fc000bc0 con 0x7f4108039660 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 -- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f4108039660 msgr2=0x7f410803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:09.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 --2- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f4108039660 0x7f410803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f410c006fd0 tx=0x7f410c006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:09.318 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 -- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 msgr2=0x7f412419c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:09.318 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 --2- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412419c680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4114011b60 tx=0x7f4114011c40 comp rx=0 tx=0).stop 2024-04-13T03:23:09.318 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 -- 172.21.15.16:0/418262200 shutdown_connections 2024-04-13T03:23:09.318 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 --2- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f4108039660 0x7f410803bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:09.318 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 --2- 172.21.15.16:0/418262200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4124109140 0x7f412419c680 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:09.318 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 -- 172.21.15.16:0/418262200 >> 172.21.15.16:0/418262200 conn(0x7f4124102b50 msgr2=0x7f4124103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:09.318 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 -- 172.21.15.16:0/418262200 shutdown_connections 2024-04-13T03:23:09.318 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:08.619+0000 7f411a000700 1 -- 172.21.15.16:0/418262200 wait complete. 2024-04-13T03:23:09.318 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-04-13T03:23:09.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:09 smithi016 ceph-mon[31772]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: Generating public/private rsa key pair. 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: Your identification has been saved in /tmp/tmp_77lmdk3/key. 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: Your public key has been saved in /tmp/tmp_77lmdk3/key.pub. 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: The key fingerprint is: 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: SHA256:Ouy6X7/6xnI1VfZ/0erbaHN+y4IUMoYCTuuHVm51M+A ceph-e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: The key's randomart image is: 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: +---[RSA 3072]----+ 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: | | 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: | o . o| 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: | o o . o oo| 2024-04-13T03:23:10.438 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: | o o E B . ..o| 2024-04-13T03:23:10.439 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: | . + o S = .. .o| 2024-04-13T03:23:10.439 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: | + = . .o . o| 2024-04-13T03:23:10.439 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: | . o + .....o .| 2024-04-13T03:23:10.439 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: | . o..+. .=oo| 2024-04-13T03:23:10.439 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: | o+o .*+. .+B=| 2024-04-13T03:23:10.439 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: +----[SHA256]-----+ 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.934+0000 7fe0d2600700 1 Processor -- start 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.935+0000 7fe0d2600700 1 -- start start 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.935+0000 7fe0d2600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc107970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.935+0000 7fe0d2600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc107f40 con 0x7fe0cc107550 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.936+0000 7fe0cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc107970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.936+0000 7fe0cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc107970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41462/0 (socket says 172.21.15.16:41462) 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.936+0000 7fe0cb400700 1 -- 172.21.15.16:0/336060015 learned_addr learned my addr 172.21.15.16:0/336060015 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.936+0000 7fe0cb400700 1 -- 172.21.15.16:0/336060015 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc108750 con 0x7fe0cc107550 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.937+0000 7fe0cb400700 1 --2- 172.21.15.16:0/336060015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc107970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0098a0 tx=0x7fe0bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4509372cabe705c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.937+0000 7fe0ca000700 1 -- 172.21.15.16:0/336060015 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc004160 con 0x7fe0cc107550 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.937+0000 7fe0ca000700 1 -- 172.21.15.16:0/336060015 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe0bc004300 con 0x7fe0cc107550 2024-04-13T03:23:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.937+0000 7fe0ca000700 1 -- 172.21.15.16:0/336060015 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc011b50 con 0x7fe0cc107550 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.938+0000 7fe0d2600700 1 -- 172.21.15.16:0/336060015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 msgr2=0x7fe0cc107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.938+0000 7fe0d2600700 1 --2- 172.21.15.16:0/336060015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc107970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0098a0 tx=0x7fe0bc009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.938+0000 7fe0d2600700 1 -- 172.21.15.16:0/336060015 shutdown_connections 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.938+0000 7fe0d2600700 1 --2- 172.21.15.16:0/336060015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc107970 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.938+0000 7fe0d2600700 1 -- 172.21.15.16:0/336060015 >> 172.21.15.16:0/336060015 conn(0x7fe0cc102b50 msgr2=0x7fe0cc104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.938+0000 7fe0d2600700 1 -- 172.21.15.16:0/336060015 shutdown_connections 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.938+0000 7fe0d2600700 1 -- 172.21.15.16:0/336060015 wait complete. 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.940+0000 7fe0d2600700 1 Processor -- start 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.940+0000 7fe0d2600700 1 -- start start 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.940+0000 7fe0d2600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.940+0000 7fe0d2600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc10bb00 con 0x7fe0cc107550 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.940+0000 7fe0cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.940+0000 7fe0cb400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41468/0 (socket says 172.21.15.16:41468) 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.941+0000 7fe0cb400700 1 -- 172.21.15.16:0/858418057 learned_addr learned my addr 172.21.15.16:0/858418057 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:11.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.941+0000 7fe0cb400700 1 -- 172.21.15.16:0/858418057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc19cc60 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.941+0000 7fe0cb400700 1 --2- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc19c720 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc011850 tx=0x7fe0bc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.941+0000 7fe0c3e00700 1 -- 172.21.15.16:0/858418057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc003af0 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.941+0000 7fe0c3e00700 1 -- 172.21.15.16:0/858418057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe0bc020070 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.942+0000 7fe0c3e00700 1 -- 172.21.15.16:0/858418057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc0195a0 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.942+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0cc19cef0 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.942+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0cc19fbe0 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.943+0000 7fe0c3e00700 1 -- 172.21.15.16:0/858418057 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe0bc019850 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.943+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0cc064110 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.943+0000 7fe0c3e00700 1 --2- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fe0b0039600 0x7fe0b003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.943+0000 7fe0c3e00700 1 -- 172.21.15.16:0/858418057 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe0bc011e00 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.944+0000 7fe0caa00700 1 --2- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fe0b0039600 0x7fe0b003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.944+0000 7fe0caa00700 1 --2- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fe0b0039600 0x7fe0b003bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8006fd0 tx=0x7fe0b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:09.949+0000 7fe0c3e00700 1 -- 172.21.15.16:0/858418057 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0bc00e260 con 0x7fe0cc107550 2024-04-13T03:23:11.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.177+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fe0cc02e280 con 0x7fe0b0039600 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.411+0000 7fe0c3e00700 1 -- 172.21.15.16:0/858418057 <== mgr.14120 v2:172.21.15.16:6800/164620940 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe0cc02e280 con 0x7fe0b0039600 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.415+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fe0b0039600 msgr2=0x7fe0b003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.415+0000 7fe0d2600700 1 --2- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fe0b0039600 0x7fe0b003bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8006fd0 tx=0x7fe0b8006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.415+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 msgr2=0x7fe0cc19c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.415+0000 7fe0d2600700 1 --2- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc19c720 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc011850 tx=0x7fe0bc011c10 comp rx=0 tx=0).stop 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.416+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 shutdown_connections 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.416+0000 7fe0d2600700 1 --2- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fe0b0039600 0x7fe0b003bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.416+0000 7fe0d2600700 1 --2- 172.21.15.16:0/858418057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe0cc107550 0x7fe0cc19c720 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.416+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 >> 172.21.15.16:0/858418057 conn(0x7fe0cc102b50 msgr2=0x7fe0cc103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.416+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 shutdown_connections 2024-04-13T03:23:11.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:10.416+0000 7fe0d2600700 1 -- 172.21.15.16:0/858418057 wait complete. 2024-04-13T03:23:11.677 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:11 smithi016 ceph-mon[31772]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:11.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:11 smithi016 ceph-mon[31772]: Generating ssh key... 2024-04-13T03:23:11.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:11 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:11.678 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:11 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:12.582 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:12 smithi016 ceph-mon[31772]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:12.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwhJeSSvHu9dhxbS/akIjigavlveQN49JXAX2H978JTMWop1/YYTpw4pbcGU6uOmtMsJZ1rz2ivC7XBOIzwfH90HrvRGJ6fM71EuA9CuQzJHer3XX51egPnC4eFb15bsMQIy8VBANlGfUTyaTAM2g3NpXdw7yBdDvDfxmeSdfsyIWzf7+I1uSTIBIZYYbG9p930pchOcnW9uBIf0vke2kN4K08s8uJa0eklcW3Vdazh+3g4kDD+v3MvGZDYFyCv86OadLRd6mIYrfDYPCHkaweyanK75PHVMpuorHZ9LSyYEBMO4RUyEn9OcbizDzBE8Qw762+OkdqI9oJPB/gFCMdI9lyhByIoYX8smteLc6fY1oP92fXaZRglsTRIy1LXuNo9su41PTVAry10IwUZpNkx0OZYxnFb8vCEmxZg7xCamxMEixcwyfDba01Oc602QbSDm0R2n1T6hxjM00fpeG+ZsWCaJG9I7xeaAo3biikwV9aDGuwHhOYE/AHaYFt1+U= ceph-e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:23:12.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.752+0000 7f6194e00700 1 Processor -- start 2024-04-13T03:23:12.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.752+0000 7f6194e00700 1 -- start start 2024-04-13T03:23:12.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.752+0000 7f6194e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619010acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:12.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.753+0000 7f6194e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f619010b280 con 0x7f619010a890 2024-04-13T03:23:12.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.753+0000 7f618de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619010acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:12.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.753+0000 7f618de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619010acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41484/0 (socket says 172.21.15.16:41484) 2024-04-13T03:23:12.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.753+0000 7f618de00700 1 -- 172.21.15.16:0/1697409691 learned_addr learned my addr 172.21.15.16:0/1697409691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:12.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.753+0000 7f618de00700 1 -- 172.21.15.16:0/1697409691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619010ba40 con 0x7f619010a890 2024-04-13T03:23:12.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.754+0000 7f618de00700 1 --2- 172.21.15.16:0/1697409691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619010acb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61800098a0 tx=0x7f6180009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c51b4265ef408f server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:12.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.754+0000 7f618ca00700 1 -- 172.21.15.16:0/1697409691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6180004160 con 0x7f619010a890 2024-04-13T03:23:12.597 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.754+0000 7f618ca00700 1 -- 172.21.15.16:0/1697409691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6180004300 con 0x7f619010a890 2024-04-13T03:23:12.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.754+0000 7f618ca00700 1 -- 172.21.15.16:0/1697409691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6180011b50 con 0x7f619010a890 2024-04-13T03:23:12.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.755+0000 7f6194e00700 1 -- 172.21.15.16:0/1697409691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 msgr2=0x7f619010acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:12.598 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.755+0000 7f6194e00700 1 --2- 172.21.15.16:0/1697409691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619010acb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61800098a0 tx=0x7f6180009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:12.599 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.755+0000 7f6194e00700 1 -- 172.21.15.16:0/1697409691 shutdown_connections 2024-04-13T03:23:12.599 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.755+0000 7f6194e00700 1 --2- 172.21.15.16:0/1697409691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619010acb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:12.599 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.755+0000 7f6194e00700 1 -- 172.21.15.16:0/1697409691 >> 172.21.15.16:0/1697409691 conn(0x7f619007de00 msgr2=0x7f61901084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:12.599 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.755+0000 7f6194e00700 1 -- 172.21.15.16:0/1697409691 shutdown_connections 2024-04-13T03:23:12.599 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.755+0000 7f6194e00700 1 -- 172.21.15.16:0/1697409691 wait complete. 2024-04-13T03:23:12.600 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.756+0000 7f6194e00700 1 Processor -- start 2024-04-13T03:23:12.600 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.756+0000 7f6194e00700 1 -- start start 2024-04-13T03:23:12.600 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.757+0000 7f6194e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:12.601 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.757+0000 7f6194e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f619010edf0 con 0x7f619010a890 2024-04-13T03:23:12.601 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.757+0000 7f618de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:12.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.757+0000 7f618de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619019e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41488/0 (socket says 172.21.15.16:41488) 2024-04-13T03:23:12.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.757+0000 7f618de00700 1 -- 172.21.15.16:0/4147638883 learned_addr learned my addr 172.21.15.16:0/4147638883 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:12.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.757+0000 7f618de00700 1 -- 172.21.15.16:0/4147638883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619019eda0 con 0x7f619010a890 2024-04-13T03:23:12.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.758+0000 7f618de00700 1 --2- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619019e860 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6180011850 tx=0x7f6180011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:12.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.758+0000 7f6186a00700 1 -- 172.21.15.16:0/4147638883 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6180003af0 con 0x7f619010a890 2024-04-13T03:23:12.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.758+0000 7f6186a00700 1 -- 172.21.15.16:0/4147638883 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6180020070 con 0x7f619010a890 2024-04-13T03:23:12.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.758+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619019f030 con 0x7f619010a890 2024-04-13T03:23:12.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.758+0000 7f6186a00700 1 -- 172.21.15.16:0/4147638883 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61800195a0 con 0x7f619010a890 2024-04-13T03:23:12.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.758+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61901a1d20 con 0x7f619010a890 2024-04-13T03:23:12.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.759+0000 7f6186a00700 1 -- 172.21.15.16:0/4147638883 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6180019850 con 0x7f619010a890 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.759+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6190064110 con 0x7f619010a890 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.760+0000 7f6186a00700 1 --2- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6174039600 0x7f617403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.760+0000 7f6186a00700 1 -- 172.21.15.16:0/4147638883 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6180011e00 con 0x7f619010a890 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.760+0000 7f618d400700 1 --2- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6174039600 0x7f617403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.761+0000 7f618d400700 1 --2- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6174039600 0x7f617403bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f617c006fd0 tx=0x7f617c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.765+0000 7f6186a00700 1 -- 172.21.15.16:0/4147638883 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f618000e260 con 0x7f619010a890 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.995+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f61901083d0 con 0x7f6174039600 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:11.997+0000 7f6186a00700 1 -- 172.21.15.16:0/4147638883 <== mgr.14120 v2:172.21.15.16:6800/164620940 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f61901083d0 con 0x7f6174039600 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6174039600 msgr2=0x7f617403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 --2- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6174039600 0x7f617403bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f617c006fd0 tx=0x7f617c006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 msgr2=0x7f619019e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 --2- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619019e860 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6180011850 tx=0x7f6180011c10 comp rx=0 tx=0).stop 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 shutdown_connections 2024-04-13T03:23:12.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 --2- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6174039600 0x7f617403bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:12.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 --2- 172.21.15.16:0/4147638883 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619010a890 0x7f619019e860 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:12.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 >> 172.21.15.16:0/4147638883 conn(0x7f619007de00 msgr2=0x7f6190107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:12.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 shutdown_connections 2024-04-13T03:23:12.606 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:12.000+0000 7f6194e00700 1 -- 172.21.15.16:0/4147638883 wait complete. 2024-04-13T03:23:12.606 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-13T03:23:12.606 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-04-13T03:23:12.608 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-04-13T03:23:14.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:14 smithi016 ceph-mon[31772]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:15.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:15 smithi016 ceph-mon[31772]: Deploying cephadm binary to smithi016 2024-04-13T03:23:20.951 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.205+0000 7f6580c00700 1 Processor -- start 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.205+0000 7f6580c00700 1 -- start start 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.206+0000 7f6580c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c10acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.206+0000 7f6580c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f657c10b2a0 con 0x7f657c10a8b0 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.206+0000 7f6579e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c10acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.206+0000 7f6579e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c10acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41504/0 (socket says 172.21.15.16:41504) 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.206+0000 7f6579e00700 1 -- 172.21.15.16:0/2677934403 learned_addr learned my addr 172.21.15.16:0/2677934403 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.207+0000 7f6579e00700 1 -- 172.21.15.16:0/2677934403 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c10ba60 con 0x7f657c10a8b0 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.207+0000 7f6579e00700 1 --2- 172.21.15.16:0/2677934403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c10acd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f656c0098a0 tx=0x7f656c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5432567a1c971b server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.208+0000 7f6578a00700 1 -- 172.21.15.16:0/2677934403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f656c004160 con 0x7f657c10a8b0 2024-04-13T03:23:20.952 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.208+0000 7f6578a00700 1 -- 172.21.15.16:0/2677934403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f656c004300 con 0x7f657c10a8b0 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.208+0000 7f6578a00700 1 -- 172.21.15.16:0/2677934403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f656c011b50 con 0x7f657c10a8b0 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.208+0000 7f6580c00700 1 -- 172.21.15.16:0/2677934403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 msgr2=0x7f657c10acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.208+0000 7f6580c00700 1 --2- 172.21.15.16:0/2677934403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c10acd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f656c0098a0 tx=0x7f656c009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.209+0000 7f6580c00700 1 -- 172.21.15.16:0/2677934403 shutdown_connections 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.209+0000 7f6580c00700 1 --2- 172.21.15.16:0/2677934403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c10acd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.209+0000 7f6580c00700 1 -- 172.21.15.16:0/2677934403 >> 172.21.15.16:0/2677934403 conn(0x7f657c07de00 msgr2=0x7f657c1084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.209+0000 7f6580c00700 1 -- 172.21.15.16:0/2677934403 shutdown_connections 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.209+0000 7f6580c00700 1 -- 172.21.15.16:0/2677934403 wait complete. 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.210+0000 7f6580c00700 1 Processor -- start 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.210+0000 7f6580c00700 1 -- start start 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.211+0000 7f6580c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.211+0000 7f6580c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f657c10ee10 con 0x7f657c10a8b0 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.211+0000 7f6579e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:20.953 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.211+0000 7f6579e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c19e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41520/0 (socket says 172.21.15.16:41520) 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.211+0000 7f6579e00700 1 -- 172.21.15.16:0/2424859058 learned_addr learned my addr 172.21.15.16:0/2424859058 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.212+0000 7f6579e00700 1 -- 172.21.15.16:0/2424859058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c19edc0 con 0x7f657c10a8b0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.212+0000 7f6579e00700 1 --2- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c19e880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f656c011850 tx=0x7f656c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.212+0000 7f6572a00700 1 -- 172.21.15.16:0/2424859058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f656c003af0 con 0x7f657c10a8b0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.212+0000 7f6572a00700 1 -- 172.21.15.16:0/2424859058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f656c020070 con 0x7f657c10a8b0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.212+0000 7f6572a00700 1 -- 172.21.15.16:0/2424859058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f656c0195a0 con 0x7f657c10a8b0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.212+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657c19f050 con 0x7f657c10a8b0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.212+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657c1a1d40 con 0x7f657c10a8b0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.213+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f657c064110 con 0x7f657c10a8b0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.214+0000 7f6572a00700 1 -- 172.21.15.16:0/2424859058 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f656c019850 con 0x7f657c10a8b0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.214+0000 7f6572a00700 1 --2- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6560039600 0x7f656003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.214+0000 7f6572a00700 1 -- 172.21.15.16:0/2424859058 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f656c011e00 con 0x7f657c10a8b0 2024-04-13T03:23:20.954 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.214+0000 7f6579400700 1 --2- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6560039600 0x7f656003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.215+0000 7f6579400700 1 --2- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6560039600 0x7f656003bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6568006fd0 tx=0x7f6568006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.220+0000 7f6572a00700 1 -- 172.21.15.16:0/2424859058 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f656c00e260 con 0x7f657c10a8b0 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:13.461+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}) -- 0x7f657c108320 con 0x7f6560039600 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.235+0000 7f6572a00700 1 -- 172.21.15.16:0/2424859058 <== mgr.14120 v2:172.21.15.16:6800/164620940 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f657c108320 con 0x7f6560039600 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.240+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6560039600 msgr2=0x7f656003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.240+0000 7f6580c00700 1 --2- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6560039600 0x7f656003bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6568006fd0 tx=0x7f6568006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.240+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 msgr2=0x7f657c19e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.240+0000 7f6580c00700 1 --2- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c19e880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f656c011850 tx=0x7f656c011c10 comp rx=0 tx=0).stop 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.240+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 shutdown_connections 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.240+0000 7f6580c00700 1 --2- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f6560039600 0x7f656003bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.240+0000 7f6580c00700 1 --2- 172.21.15.16:0/2424859058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f657c10a8b0 0x7f657c19e880 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.240+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 >> 172.21.15.16:0/2424859058 conn(0x7f657c07de00 msgr2=0x7f657c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.241+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 shutdown_connections 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:20.241+0000 7f6580c00700 1 -- 172.21.15.16:0/2424859058 wait complete. 2024-04-13T03:23:20.955 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mon service... 2024-04-13T03:23:21.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:21 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:21.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:21 smithi016 ceph-mon[31772]: Added host smithi016 2024-04-13T03:23:21.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:21 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:22.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.594+0000 7f9881800700 1 Processor -- start 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.594+0000 7f9881800700 1 -- start start 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.595+0000 7f9881800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c10acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.595+0000 7f9881800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f987c10b280 con 0x7f987c10a890 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.595+0000 7f987aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c10acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.595+0000 7f987aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c10acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50614/0 (socket says 172.21.15.16:50614) 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.595+0000 7f987aa00700 1 -- 172.21.15.16:0/4216697276 learned_addr learned my addr 172.21.15.16:0/4216697276 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.596+0000 7f987aa00700 1 -- 172.21.15.16:0/4216697276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c10ba40 con 0x7f987c10a890 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.596+0000 7f987aa00700 1 --2- 172.21.15.16:0/4216697276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c10acb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f986c0098a0 tx=0x7f986c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e9ebb9c941198e server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.596+0000 7f9879600700 1 -- 172.21.15.16:0/4216697276 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f986c004160 con 0x7f987c10a890 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.596+0000 7f9879600700 1 -- 172.21.15.16:0/4216697276 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f986c004300 con 0x7f987c10a890 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.596+0000 7f9879600700 1 -- 172.21.15.16:0/4216697276 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f986c011b50 con 0x7f987c10a890 2024-04-13T03:23:22.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.597+0000 7f9881800700 1 -- 172.21.15.16:0/4216697276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 msgr2=0x7f987c10acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.597+0000 7f9881800700 1 --2- 172.21.15.16:0/4216697276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c10acb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f986c0098a0 tx=0x7f986c009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.597+0000 7f9881800700 1 -- 172.21.15.16:0/4216697276 shutdown_connections 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.597+0000 7f9881800700 1 --2- 172.21.15.16:0/4216697276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c10acb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.597+0000 7f9881800700 1 -- 172.21.15.16:0/4216697276 >> 172.21.15.16:0/4216697276 conn(0x7f987c07de00 msgr2=0x7f987c1084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.597+0000 7f9881800700 1 -- 172.21.15.16:0/4216697276 shutdown_connections 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.597+0000 7f9881800700 1 -- 172.21.15.16:0/4216697276 wait complete. 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.598+0000 7f9881800700 1 Processor -- start 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.598+0000 7f9881800700 1 -- start start 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.599+0000 7f9881800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.599+0000 7f9881800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f987c10edf0 con 0x7f987c10a890 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.599+0000 7f987aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.599+0000 7f987aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c19e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50624/0 (socket says 172.21.15.16:50624) 2024-04-13T03:23:22.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.599+0000 7f987aa00700 1 -- 172.21.15.16:0/1066910938 learned_addr learned my addr 172.21.15.16:0/1066910938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.599+0000 7f987aa00700 1 -- 172.21.15.16:0/1066910938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c19eda0 con 0x7f987c10a890 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.600+0000 7f987aa00700 1 --2- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c19e860 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f986c009fb0 tx=0x7f986c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.600+0000 7f9873400700 1 -- 172.21.15.16:0/1066910938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f986c0038b0 con 0x7f987c10a890 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.600+0000 7f9873400700 1 -- 172.21.15.16:0/1066910938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f986c011e50 con 0x7f987c10a890 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.600+0000 7f9873400700 1 -- 172.21.15.16:0/1066910938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f986c019550 con 0x7f987c10a890 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.600+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987c19f030 con 0x7f987c10a890 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.600+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987c1a1d20 con 0x7f987c10a890 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.601+0000 7f9873400700 1 -- 172.21.15.16:0/1066910938 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f986c003a50 con 0x7f987c10a890 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.602+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f987c064110 con 0x7f987c10a890 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.602+0000 7f9873400700 1 --2- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f98600395b0 0x7f986003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.602+0000 7f9873400700 1 -- 172.21.15.16:0/1066910938 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f986c04d8f0 con 0x7f987c10a890 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.602+0000 7f987a000700 1 --2- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f98600395b0 0x7f986003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.603+0000 7f987a000700 1 --2- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f98600395b0 0x7f986003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9868006fd0 tx=0x7f9868006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:22.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.608+0000 7f9873400700 1 -- 172.21.15.16:0/1066910938 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f986c020240 con 0x7f987c10a890 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.867+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f987c1083d0 con 0x7f98600395b0 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.878+0000 7f9873400700 1 -- 172.21.15.16:0/1066910938 <== mgr.14120 v2:172.21.15.16:6800/164620940 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f987c1083d0 con 0x7f98600395b0 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f98600395b0 msgr2=0x7f986003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 --2- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f98600395b0 0x7f986003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9868006fd0 tx=0x7f9868006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 msgr2=0x7f987c19e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 --2- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c19e860 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f986c009fb0 tx=0x7f986c011bf0 comp rx=0 tx=0).stop 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 shutdown_connections 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 --2- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f98600395b0 0x7f986003ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 --2- 172.21.15.16:0/1066910938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f987c10a890 0x7f987c19e860 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 >> 172.21.15.16:0/1066910938 conn(0x7f987c07de00 msgr2=0x7f987c107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 shutdown_connections 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:21.881+0000 7f9881800700 1 -- 172.21.15.16:0/1066910938 wait complete. 2024-04-13T03:23:22.596 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mgr service... 2024-04-13T03:23:23.129 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:22 smithi016 ceph-mon[31772]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:23.130 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:22 smithi016 ceph-mon[31772]: Saving service mon spec with placement count:5 2024-04-13T03:23:23.130 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:22 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:24.746 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-13T03:23:24.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.602+0000 7f073be00700 1 Processor -- start 2024-04-13T03:23:24.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.603+0000 7f073be00700 1 -- start start 2024-04-13T03:23:24.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.603+0000 7f073be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f0734108a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:24.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.603+0000 7f073be00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0734109020 con 0x7f0734108630 2024-04-13T03:23:24.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.604+0000 7f0739400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f0734108a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:24.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.604+0000 7f0739400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f0734108a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50626/0 (socket says 172.21.15.16:50626) 2024-04-13T03:23:24.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.604+0000 7f0739400700 1 -- 172.21.15.16:0/528356096 learned_addr learned my addr 172.21.15.16:0/528356096 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:24.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.604+0000 7f0739400700 1 -- 172.21.15.16:0/528356096 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07341097e0 con 0x7f0734108630 2024-04-13T03:23:24.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.604+0000 7f0739400700 1 --2- 172.21.15.16:0/528356096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f0734108a50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f07280098a0 tx=0x7f0728009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed5b21023e50016 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:24.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.605+0000 7f0733e00700 1 -- 172.21.15.16:0/528356096 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0728004160 con 0x7f0734108630 2024-04-13T03:23:24.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.605+0000 7f0733e00700 1 -- 172.21.15.16:0/528356096 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0728004300 con 0x7f0734108630 2024-04-13T03:23:24.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.605+0000 7f0733e00700 1 -- 172.21.15.16:0/528356096 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0728011b50 con 0x7f0734108630 2024-04-13T03:23:24.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.606+0000 7f073be00700 1 -- 172.21.15.16:0/528356096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 msgr2=0x7f0734108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:24.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.606+0000 7f073be00700 1 --2- 172.21.15.16:0/528356096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f0734108a50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f07280098a0 tx=0x7f0728009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:24.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.606+0000 7f073be00700 1 -- 172.21.15.16:0/528356096 shutdown_connections 2024-04-13T03:23:24.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.606+0000 7f073be00700 1 --2- 172.21.15.16:0/528356096 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f0734108a50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:24.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.606+0000 7f073be00700 1 -- 172.21.15.16:0/528356096 >> 172.21.15.16:0/528356096 conn(0x7f0734103c50 msgr2=0x7f0734106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:24.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.606+0000 7f073be00700 1 -- 172.21.15.16:0/528356096 shutdown_connections 2024-04-13T03:23:24.753 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.606+0000 7f073be00700 1 -- 172.21.15.16:0/528356096 wait complete. 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.607+0000 7f073be00700 1 Processor -- start 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.607+0000 7f073be00700 1 -- start start 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.607+0000 7f073be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f073419c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.607+0000 7f073be00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f073410cb90 con 0x7f0734108630 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.608+0000 7f0739400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f073419c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.608+0000 7f0739400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f073419c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50640/0 (socket says 172.21.15.16:50640) 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.608+0000 7f0739400700 1 -- 172.21.15.16:0/1756019021 learned_addr learned my addr 172.21.15.16:0/1756019021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.608+0000 7f0739400700 1 -- 172.21.15.16:0/1756019021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073419cb40 con 0x7f0734108630 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.609+0000 7f0739400700 1 --2- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f073419c600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0728000c00 tx=0x7f0728011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.609+0000 7f0732000700 1 -- 172.21.15.16:0/1756019021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0728003b20 con 0x7f0734108630 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.609+0000 7f0732000700 1 -- 172.21.15.16:0/1756019021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0728003cc0 con 0x7f0734108630 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.609+0000 7f0732000700 1 -- 172.21.15.16:0/1756019021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0728024910 con 0x7f0734108630 2024-04-13T03:23:24.754 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.609+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073419cdd0 con 0x7f0734108630 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.609+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f073419fac0 con 0x7f0734108630 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.610+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0734064110 con 0x7f0734108630 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.611+0000 7f0732000700 1 -- 172.21.15.16:0/1756019021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0728022020 con 0x7f0734108630 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.611+0000 7f0732000700 1 --2- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f071c039600 0x7f071c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.611+0000 7f0732000700 1 -- 172.21.15.16:0/1756019021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0728011e00 con 0x7f0734108630 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.611+0000 7f0738a00700 1 --2- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f071c039600 0x7f071c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.612+0000 7f0738a00700 1 --2- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f071c039600 0x7f071c03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0720006fd0 tx=0x7f0720006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.617+0000 7f0732000700 1 -- 172.21.15.16:0/1756019021 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f072801ad80 con 0x7f0734108630 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.866+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f0734104f00 con 0x7f071c039600 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.876+0000 7f0732000700 1 -- 172.21.15.16:0/1756019021 <== mgr.14120 v2:172.21.15.16:6800/164620940 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0734104f00 con 0x7f071c039600 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.879+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f071c039600 msgr2=0x7f071c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.879+0000 7f073be00700 1 --2- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f071c039600 0x7f071c03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0720006fd0 tx=0x7f0720006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:24.755 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.879+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 msgr2=0x7f073419c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:24.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.879+0000 7f073be00700 1 --2- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f073419c600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0728000c00 tx=0x7f0728011c10 comp rx=0 tx=0).stop 2024-04-13T03:23:24.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.879+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 shutdown_connections 2024-04-13T03:23:24.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.879+0000 7f073be00700 1 --2- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f071c039600 0x7f071c03bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:24.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.879+0000 7f073be00700 1 --2- 172.21.15.16:0/1756019021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0734108630 0x7f073419c600 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:24.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.879+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 >> 172.21.15.16:0/1756019021 conn(0x7f0734103c50 msgr2=0x7f07341047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:24.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.880+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 shutdown_connections 2024-04-13T03:23:24.756 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:23.880+0000 7f073be00700 1 -- 172.21.15.16:0/1756019021 wait complete. 2024-04-13T03:23:25.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:24 smithi016 ceph-mon[31772]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:25.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:24 smithi016 ceph-mon[31772]: Saving service mgr spec with placement count:2 2024-04-13T03:23:25.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:24 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:25.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:24 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:26.046 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:25 smithi016 ceph-mon[31772]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:23:26.046 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:25 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:26.046 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:25 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2133061014' entity='client.admin' 2024-04-13T03:23:26.387 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.367+0000 7f8929a00700 1 Processor -- start 2024-04-13T03:23:26.387 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.367+0000 7f8929a00700 1 -- start start 2024-04-13T03:23:26.387 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.368+0000 7f8929a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f8924108ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:26.387 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.368+0000 7f8929a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8924109080 con 0x7f8924108690 2024-04-13T03:23:26.387 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.368+0000 7f8922a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f8924108ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.368+0000 7f8922a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f8924108ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50642/0 (socket says 172.21.15.16:50642) 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.368+0000 7f8922a00700 1 -- 172.21.15.16:0/4002849120 learned_addr learned my addr 172.21.15.16:0/4002849120 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.369+0000 7f8922a00700 1 -- 172.21.15.16:0/4002849120 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89241097f0 con 0x7f8924108690 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.369+0000 7f8922a00700 1 --2- 172.21.15.16:0/4002849120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f8924108ab0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f89140098a0 tx=0x7f8914009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d676530e0ab79885 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.370+0000 7f8921600700 1 -- 172.21.15.16:0/4002849120 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8914004160 con 0x7f8924108690 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.370+0000 7f8921600700 1 -- 172.21.15.16:0/4002849120 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8914004300 con 0x7f8924108690 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.370+0000 7f8921600700 1 -- 172.21.15.16:0/4002849120 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8914011b20 con 0x7f8924108690 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.370+0000 7f8929a00700 1 -- 172.21.15.16:0/4002849120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 msgr2=0x7f8924108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.370+0000 7f8929a00700 1 --2- 172.21.15.16:0/4002849120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f8924108ab0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f89140098a0 tx=0x7f8914009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.371+0000 7f8929a00700 1 -- 172.21.15.16:0/4002849120 shutdown_connections 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.371+0000 7f8929a00700 1 --2- 172.21.15.16:0/4002849120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f8924108ab0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.371+0000 7f8929a00700 1 -- 172.21.15.16:0/4002849120 >> 172.21.15.16:0/4002849120 conn(0x7f8924103cb0 msgr2=0x7f89241060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.371+0000 7f8929a00700 1 -- 172.21.15.16:0/4002849120 shutdown_connections 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.371+0000 7f8929a00700 1 -- 172.21.15.16:0/4002849120 wait complete. 2024-04-13T03:23:26.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.372+0000 7f8929a00700 1 Processor -- start 2024-04-13T03:23:26.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.372+0000 7f8929a00700 1 -- start start 2024-04-13T03:23:26.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.372+0000 7f8929a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f892407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:26.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.372+0000 7f8929a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f892410cba0 con 0x7f8924108690 2024-04-13T03:23:26.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.373+0000 7f8922a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f892407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:26.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.373+0000 7f8922a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f892407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50650/0 (socket says 172.21.15.16:50650) 2024-04-13T03:23:26.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.373+0000 7f8922a00700 1 -- 172.21.15.16:0/2133061014 learned_addr learned my addr 172.21.15.16:0/2133061014 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:26.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.373+0000 7f8922a00700 1 -- 172.21.15.16:0/2133061014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892407e0a0 con 0x7f8924108690 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.374+0000 7f8922a00700 1 --2- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f892407db60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89140117e0 tx=0x7f8914011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.374+0000 7f891b400700 1 -- 172.21.15.16:0/2133061014 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8914003bf0 con 0x7f8924108690 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.374+0000 7f891b400700 1 -- 172.21.15.16:0/2133061014 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f891401a910 con 0x7f8924108690 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.374+0000 7f891b400700 1 -- 172.21.15.16:0/2133061014 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8914024970 con 0x7f8924108690 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.374+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892407c290 con 0x7f8924108690 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.374+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f892407c750 con 0x7f8924108690 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.375+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8924064110 con 0x7f8924108690 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.376+0000 7f891b400700 1 -- 172.21.15.16:0/2133061014 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f8914022020 con 0x7f8924108690 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.376+0000 7f891b400700 1 --2- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f8908039600 0x7f890803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.376+0000 7f891b400700 1 -- 172.21.15.16:0/2133061014 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f891404dad0 con 0x7f8924108690 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.377+0000 7f8922000700 1 --2- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f8908039600 0x7f890803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:26.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.377+0000 7f8922000700 1 --2- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f8908039600 0x7f890803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8910006fd0 tx=0x7f8910006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.382+0000 7f891b400700 1 -- 172.21.15.16:0/2133061014 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f891404d420 con 0x7f8924108690 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.627+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f892407ce10 con 0x7f8924108690 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.631+0000 7f891b400700 1 -- 172.21.15.16:0/2133061014 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f891401ab70 con 0x7f8924108690 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.634+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f8908039600 msgr2=0x7f890803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.634+0000 7f8929a00700 1 --2- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f8908039600 0x7f890803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8910006fd0 tx=0x7f8910006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.634+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 msgr2=0x7f892407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.634+0000 7f8929a00700 1 --2- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f892407db60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89140117e0 tx=0x7f8914011ba0 comp rx=0 tx=0).stop 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.634+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 shutdown_connections 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.635+0000 7f8929a00700 1 --2- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f8908039600 0x7f890803bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.635+0000 7f8929a00700 1 --2- 172.21.15.16:0/2133061014 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8924108690 0x7f892407db60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.635+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 >> 172.21.15.16:0/2133061014 conn(0x7f8924103cb0 msgr2=0x7f89241047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.635+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 shutdown_connections 2024-04-13T03:23:26.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:25.635+0000 7f8929a00700 1 -- 172.21.15.16:0/2133061014 wait complete. 2024-04-13T03:23:28.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:28 smithi016 ceph-mon[31772]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:23:28.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:28 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/910382455' entity='client.admin' 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.032+0000 7f77d3a00700 1 Processor -- start 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.033+0000 7f77d3a00700 1 -- start start 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.033+0000 7f77d3a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc10f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.033+0000 7f77d3a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc10f960 con 0x7f77cc10f000 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.034+0000 7f77d1000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc10f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.034+0000 7f77d1000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc10f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50654/0 (socket says 172.21.15.16:50654) 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.034+0000 7f77d1000700 1 -- 172.21.15.16:0/737193119 learned_addr learned my addr 172.21.15.16:0/737193119 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.034+0000 7f77d1000700 1 -- 172.21.15.16:0/737193119 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77cc10fae0 con 0x7f77cc10f000 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.035+0000 7f77d1000700 1 --2- 172.21.15.16:0/737193119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc10f420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009e80 tx=0x7f77c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2153718dd53ae0e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.035+0000 7f77cb400700 1 -- 172.21.15.16:0/737193119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c0017070 con 0x7f77cc10f000 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.035+0000 7f77cb400700 1 -- 172.21.15.16:0/737193119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c000b8c0 con 0x7f77cc10f000 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.035+0000 7f77cb400700 1 -- 172.21.15.16:0/737193119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c00045c0 con 0x7f77cc10f000 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.036+0000 7f77d3a00700 1 -- 172.21.15.16:0/737193119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 msgr2=0x7f77cc10f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.036+0000 7f77d3a00700 1 --2- 172.21.15.16:0/737193119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc10f420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009e80 tx=0x7f77c000b330 comp rx=0 tx=0).stop 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.036+0000 7f77d3a00700 1 -- 172.21.15.16:0/737193119 shutdown_connections 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.036+0000 7f77d3a00700 1 --2- 172.21.15.16:0/737193119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc10f420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.036+0000 7f77d3a00700 1 -- 172.21.15.16:0/737193119 >> 172.21.15.16:0/737193119 conn(0x7f77cc07de00 msgr2=0x7f77cc080220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.036+0000 7f77d3a00700 1 -- 172.21.15.16:0/737193119 shutdown_connections 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.036+0000 7f77d3a00700 1 -- 172.21.15.16:0/737193119 wait complete. 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.037+0000 7f77d3a00700 1 Processor -- start 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.037+0000 7f77d3a00700 1 -- start start 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.038+0000 7f77d3a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc1a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.038+0000 7f77d3a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc113090 con 0x7f77cc10f000 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.038+0000 7f77d1000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc1a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.038+0000 7f77d1000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc1a2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50668/0 (socket says 172.21.15.16:50668) 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.038+0000 7f77d1000700 1 -- 172.21.15.16:0/910382455 learned_addr learned my addr 172.21.15.16:0/910382455 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.038+0000 7f77d1000700 1 -- 172.21.15.16:0/910382455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77cc1a30b0 con 0x7f77cc10f000 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.039+0000 7f77d1000700 1 --2- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc1a2b70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009fd0 tx=0x7f77c0003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.039+0000 7f77c9600700 1 -- 172.21.15.16:0/910382455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c0020020 con 0x7f77cc10f000 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.039+0000 7f77c9600700 1 -- 172.21.15.16:0/910382455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c000bba0 con 0x7f77cc10f000 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.039+0000 7f77c9600700 1 -- 172.21.15.16:0/910382455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c0017040 con 0x7f77cc10f000 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.039+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77cc1a3340 con 0x7f77cc10f000 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.039+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77cc1a5fe0 con 0x7f77cc10f000 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.041+0000 7f77c9600700 1 -- 172.21.15.16:0/910382455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f77c0023020 con 0x7f77cc10f000 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.041+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77cc113210 con 0x7f77cc10f000 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.041+0000 7f77c9600700 1 --2- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f77b40395b0 0x7f77b403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.041+0000 7f77c9600700 1 -- 172.21.15.16:0/910382455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f77c000f050 con 0x7f77cc10f000 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.043+0000 7f77cbe00700 1 --2- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f77b40395b0 0x7f77b403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.043+0000 7f77cbe00700 1 --2- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f77b40395b0 0x7f77b403ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f77b8006fd0 tx=0x7f77b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.051+0000 7f77c9600700 1 -- 172.21.15.16:0/910382455 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77c002a020 con 0x7f77cc10f000 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.298+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f77cc1a61b0 con 0x7f77cc10f000 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.303+0000 7f77c9600700 1 -- 172.21.15.16:0/910382455 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f77c002b260 con 0x7f77cc10f000 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.306+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f77b40395b0 msgr2=0x7f77b403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.306+0000 7f77d3a00700 1 --2- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f77b40395b0 0x7f77b403ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f77b8006fd0 tx=0x7f77b8006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.306+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 msgr2=0x7f77cc1a2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.306+0000 7f77d3a00700 1 --2- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc1a2b70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f77c0009fd0 tx=0x7f77c0003d00 comp rx=0 tx=0).stop 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.306+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 shutdown_connections 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.306+0000 7f77d3a00700 1 --2- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f77b40395b0 0x7f77b403ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.306+0000 7f77d3a00700 1 --2- 172.21.15.16:0/910382455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77cc10f000 0x7f77cc1a2b70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.306+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 >> 172.21.15.16:0/910382455 conn(0x7f77cc07de00 msgr2=0x7f77cc07e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.307+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 shutdown_connections 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:27.307+0000 7f77d3a00700 1 -- 172.21.15.16:0/910382455 wait complete. 2024-04-13T03:23:28.578 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-04-13T03:23:30.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:29 smithi016 ceph-mon[31772]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:23:30.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:29 smithi016 ceph-mon[31772]: from='mgr.14120 172.21.15.16:0/3773912197' entity='mgr.x' 2024-04-13T03:23:30.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:29 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2181313711' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-13T03:23:30.604 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:30 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: ignoring --setuser ceph since I am not root 2024-04-13T03:23:30.605 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:30 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: ignoring --setgroup ceph since I am not root 2024-04-13T03:23:31.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.171+0000 7f497da00700 1 Processor -- start 2024-04-13T03:23:31.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.172+0000 7f497da00700 1 -- start start 2024-04-13T03:23:31.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.172+0000 7f497da00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f4978108a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:31.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.172+0000 7f497da00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4978109020 con 0x7f4978108630 2024-04-13T03:23:31.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.173+0000 7f4976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f4978108a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:31.186 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.173+0000 7f4976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f4978108a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51788/0 (socket says 172.21.15.16:51788) 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.173+0000 7f4976a00700 1 -- 172.21.15.16:0/4084594210 learned_addr learned my addr 172.21.15.16:0/4084594210 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.173+0000 7f4976a00700 1 -- 172.21.15.16:0/4084594210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49781097e0 con 0x7f4978108630 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.174+0000 7f4976a00700 1 --2- 172.21.15.16:0/4084594210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f4978108a50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f49680098a0 tx=0x7f4968009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad384d37e1e0366 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.174+0000 7f4975600700 1 -- 172.21.15.16:0/4084594210 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4968004160 con 0x7f4978108630 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.174+0000 7f4975600700 1 -- 172.21.15.16:0/4084594210 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4968004300 con 0x7f4978108630 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.174+0000 7f4975600700 1 -- 172.21.15.16:0/4084594210 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4968011b20 con 0x7f4978108630 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.175+0000 7f497da00700 1 -- 172.21.15.16:0/4084594210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 msgr2=0x7f4978108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.175+0000 7f497da00700 1 --2- 172.21.15.16:0/4084594210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f4978108a50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f49680098a0 tx=0x7f4968009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.175+0000 7f497da00700 1 -- 172.21.15.16:0/4084594210 shutdown_connections 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.175+0000 7f497da00700 1 --2- 172.21.15.16:0/4084594210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f4978108a50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.175+0000 7f497da00700 1 -- 172.21.15.16:0/4084594210 >> 172.21.15.16:0/4084594210 conn(0x7f4978103c50 msgr2=0x7f4978106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.175+0000 7f497da00700 1 -- 172.21.15.16:0/4084594210 shutdown_connections 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.175+0000 7f497da00700 1 -- 172.21.15.16:0/4084594210 wait complete. 2024-04-13T03:23:31.187 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.177+0000 7f497da00700 1 Processor -- start 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.177+0000 7f497da00700 1 -- start start 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.177+0000 7f497da00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f497819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.177+0000 7f497da00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f497810cb90 con 0x7f4978108630 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.178+0000 7f4976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f497819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.178+0000 7f4976a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f497819e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51790/0 (socket says 172.21.15.16:51790) 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.178+0000 7f4976a00700 1 -- 172.21.15.16:0/2181313711 learned_addr learned my addr 172.21.15.16:0/2181313711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.178+0000 7f4976a00700 1 -- 172.21.15.16:0/2181313711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497819ecc0 con 0x7f4978108630 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.178+0000 7f4976a00700 1 --2- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f497819e780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f49680117e0 tx=0x7f4968011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.179+0000 7f496f400700 1 -- 172.21.15.16:0/2181313711 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4968003bf0 con 0x7f4978108630 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.179+0000 7f496f400700 1 -- 172.21.15.16:0/2181313711 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f496801a910 con 0x7f4978108630 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.179+0000 7f496f400700 1 -- 172.21.15.16:0/2181313711 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4968024970 con 0x7f4978108630 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.179+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497819ef50 con 0x7f4978108630 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.179+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497819f3b0 con 0x7f4978108630 2024-04-13T03:23:31.188 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.180+0000 7f496f400700 1 -- 172.21.15.16:0/2181313711 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4968022020 con 0x7f4978108630 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.180+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4978064110 con 0x7f4978108630 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.181+0000 7f496f400700 1 --2- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f495c039600 0x7f495c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.181+0000 7f496f400700 1 -- 172.21.15.16:0/2181313711 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f496804dab0 con 0x7f4978108630 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.181+0000 7f4976000700 1 --2- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f495c039600 0x7f495c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.182+0000 7f4976000700 1 --2- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f495c039600 0x7f495c03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4964006fd0 tx=0x7f4964006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.187+0000 7f496f400700 1 -- 172.21.15.16:0/2181313711 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49680212c0 con 0x7f4978108630 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:29.482+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f49781a1d80 con 0x7f4978108630 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.304+0000 7f496f400700 1 -- 172.21.15.16:0/2181313711 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f496804d440 con 0x7f4978108630 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.309+0000 7f496f400700 1 -- 172.21.15.16:0/2181313711 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f496802d3f0 con 0x7f4978108630 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.309+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f495c039600 msgr2=0x7f495c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.309+0000 7f497da00700 1 --2- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f495c039600 0x7f495c03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4964006fd0 tx=0x7f4964006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.309+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 msgr2=0x7f497819e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:31.189 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.309+0000 7f497da00700 1 --2- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f497819e780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f49680117e0 tx=0x7f4968011ba0 comp rx=0 tx=0).stop 2024-04-13T03:23:31.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.309+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 shutdown_connections 2024-04-13T03:23:31.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.309+0000 7f497da00700 1 --2- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f495c039600 0x7f495c03bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:31.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.309+0000 7f497da00700 1 --2- 172.21.15.16:0/2181313711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4978108630 0x7f497819e780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:31.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.309+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 >> 172.21.15.16:0/2181313711 conn(0x7f4978103c50 msgr2=0x7f4978104730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:31.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.310+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 shutdown_connections 2024-04-13T03:23:31.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:30.310+0000 7f497da00700 1 -- 172.21.15.16:0/2181313711 wait complete. 2024-04-13T03:23:31.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:31 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2181313711' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-13T03:23:31.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:31 smithi016 ceph-mon[31772]: mgrmap e9: x(active, since 25s) 2024-04-13T03:23:31.478 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:31.186+0000 7f7ae777a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T03:23:31.478 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:31.326+0000 7f7ae777a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T03:23:32.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:32 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/573151698' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-13T03:23:32.355 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:32.176+0000 7f7ae777a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T03:23:32.355 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:32.303+0000 7f7ae777a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T03:23:32.746 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:32.495+0000 7f7ae777a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T03:23:32.746 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:32.626+0000 7f7ae777a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T03:23:32.746 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:32.742+0000 7f7ae777a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T03:23:32.798 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-13T03:23:32.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-13T03:23:32.799 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-13T03:23:32.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-13T03:23:32.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-13T03:23:32.800 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-13T03:23:32.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.771+0000 7fdaa3e00700 1 Processor -- start 2024-04-13T03:23:32.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.771+0000 7fdaa3e00700 1 -- start start 2024-04-13T03:23:32.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.771+0000 7fdaa3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c10b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:32.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.772+0000 7fdaa3e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c07be20 con 0x7fda9c109160 2024-04-13T03:23:32.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.772+0000 7fdaa1400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c10b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:32.801 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.772+0000 7fdaa1400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c10b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51830/0 (socket says 172.21.15.16:51830) 2024-04-13T03:23:32.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.772+0000 7fdaa1400700 1 -- 172.21.15.16:0/2881028149 learned_addr learned my addr 172.21.15.16:0/2881028149 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:32.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.773+0000 7fdaa1400700 1 -- 172.21.15.16:0/2881028149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c10ba90 con 0x7fda9c109160 2024-04-13T03:23:32.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.773+0000 7fdaa1400700 1 --2- 172.21.15.16:0/2881028149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c10b550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fda90009e80 tx=0x7fda9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a2e25f68ee09e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:32.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.773+0000 7fda9be00700 1 -- 172.21.15.16:0/2881028149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda90017070 con 0x7fda9c109160 2024-04-13T03:23:32.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.773+0000 7fda9be00700 1 -- 172.21.15.16:0/2881028149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda9000b8c0 con 0x7fda9c109160 2024-04-13T03:23:32.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.773+0000 7fda9be00700 1 -- 172.21.15.16:0/2881028149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda900045c0 con 0x7fda9c109160 2024-04-13T03:23:32.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.774+0000 7fdaa3e00700 1 -- 172.21.15.16:0/2881028149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 msgr2=0x7fda9c10b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:32.803 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.774+0000 7fdaa3e00700 1 --2- 172.21.15.16:0/2881028149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c10b550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fda90009e80 tx=0x7fda9000b330 comp rx=0 tx=0).stop 2024-04-13T03:23:32.804 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.774+0000 7fdaa3e00700 1 -- 172.21.15.16:0/2881028149 shutdown_connections 2024-04-13T03:23:32.804 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.774+0000 7fdaa3e00700 1 --2- 172.21.15.16:0/2881028149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c10b550 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:32.804 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.774+0000 7fdaa3e00700 1 -- 172.21.15.16:0/2881028149 >> 172.21.15.16:0/2881028149 conn(0x7fda9c102c10 msgr2=0x7fda9c105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:32.804 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.774+0000 7fdaa3e00700 1 -- 172.21.15.16:0/2881028149 shutdown_connections 2024-04-13T03:23:32.804 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.774+0000 7fdaa3e00700 1 -- 172.21.15.16:0/2881028149 wait complete. 2024-04-13T03:23:32.804 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.776+0000 7fdaa3e00700 1 Processor -- start 2024-04-13T03:23:32.805 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.776+0000 7fdaa3e00700 1 -- start start 2024-04-13T03:23:32.806 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.777+0000 7fdaa3e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c19e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:32.806 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.777+0000 7fdaa3e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c10ca90 con 0x7fda9c109160 2024-04-13T03:23:32.806 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.777+0000 7fdaa1400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c19e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:32.806 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.777+0000 7fdaa1400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c19e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51834/0 (socket says 172.21.15.16:51834) 2024-04-13T03:23:32.806 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.777+0000 7fdaa1400700 1 -- 172.21.15.16:0/573151698 learned_addr learned my addr 172.21.15.16:0/573151698 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:32.806 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.777+0000 7fdaa1400700 1 -- 172.21.15.16:0/573151698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c19ed80 con 0x7fda9c109160 2024-04-13T03:23:32.806 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.778+0000 7fdaa1400700 1 --2- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c19e840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fda90009530 tx=0x7fda90003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:32.806 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.778+0000 7fda9a000700 1 -- 172.21.15.16:0/573151698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda90020020 con 0x7fda9c109160 2024-04-13T03:23:32.807 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.778+0000 7fda9a000700 1 -- 172.21.15.16:0/573151698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda9000bba0 con 0x7fda9c109160 2024-04-13T03:23:32.808 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.778+0000 7fda9a000700 1 -- 172.21.15.16:0/573151698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda90017040 con 0x7fda9c109160 2024-04-13T03:23:32.808 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.778+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c19f010 con 0x7fda9c109160 2024-04-13T03:23:32.808 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.778+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c1a1d00 con 0x7fda9c109160 2024-04-13T03:23:32.808 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.779+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c064110 con 0x7fda9c109160 2024-04-13T03:23:32.808 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.779+0000 7fda9a000700 1 -- 172.21.15.16:0/573151698 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fda90023020 con 0x7fda9c109160 2024-04-13T03:23:32.808 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.780+0000 7fda9a000700 1 --2- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fda84039660 0x7fda8403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:32.808 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.780+0000 7fda9a000700 1 -- 172.21.15.16:0/573151698 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fda9000f050 con 0x7fda9c109160 2024-04-13T03:23:32.808 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.780+0000 7fdaa0a00700 1 -- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fda84039660 msgr2=0x7fda8403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/164620940 2024-04-13T03:23:32.809 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.780+0000 7fdaa0a00700 1 --2- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fda84039660 0x7fda8403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:23:32.810 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.785+0000 7fda9a000700 1 -- 172.21.15.16:0/573151698 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda9002a020 con 0x7fda9c109160 2024-04-13T03:23:32.810 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.981+0000 7fdaa0a00700 1 -- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fda84039660 msgr2=0x7fda8403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/164620940 2024-04-13T03:23:32.810 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:31.981+0000 7fdaa0a00700 1 --2- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fda84039660 0x7fda8403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-13T03:23:32.810 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.075+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fda9c1a1ed0 con 0x7fda9c109160 2024-04-13T03:23:32.810 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.076+0000 7fda9a000700 1 -- 172.21.15.16:0/573151698 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fda9002d350 con 0x7fda9c109160 2024-04-13T03:23:32.810 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.078+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fda84039660 msgr2=0x7fda8403bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:23:32.810 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.078+0000 7fdaa3e00700 1 --2- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fda84039660 0x7fda8403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:32.810 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.078+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 msgr2=0x7fda9c19e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:32.810 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.078+0000 7fdaa3e00700 1 --2- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c19e840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fda90009530 tx=0x7fda90003d00 comp rx=0 tx=0).stop 2024-04-13T03:23:32.811 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.079+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 shutdown_connections 2024-04-13T03:23:32.811 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.079+0000 7fdaa3e00700 1 --2- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7fda84039660 0x7fda8403bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:32.811 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.079+0000 7fdaa3e00700 1 --2- 172.21.15.16:0/573151698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c109160 0x7fda9c19e840 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:32.811 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.079+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 >> 172.21.15.16:0/573151698 conn(0x7fda9c102c10 msgr2=0x7fda9c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:32.811 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.079+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 shutdown_connections 2024-04-13T03:23:32.812 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:32.079+0000 7fdaa3e00700 1 -- 172.21.15.16:0/573151698 wait complete. 2024-04-13T03:23:32.812 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-04-13T03:23:32.812 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-04-13T03:23:33.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:33.116+0000 7f7ae777a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T03:23:33.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:33.264+0000 7f7ae777a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T03:23:33.727 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:33.385+0000 7f7ae777a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T03:23:33.728 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:33.519+0000 7f7ae777a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T03:23:34.104 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:33.749+0000 7f7ae777a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T03:23:35.285 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:35 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:35.100+0000 7f7ae777a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T03:23:35.285 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:35 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:35.283+0000 7f7ae777a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T03:23:36.104 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:35 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:35.726+0000 7f7ae777a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T03:23:36.104 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:35 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:35.852+0000 7f7ae777a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T03:23:36.382 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:36.121+0000 7f7ae777a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:23:36.382 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:36.380+0000 7f7ae777a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T03:23:37.104 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:36.779+0000 7f7ae777a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:23:37.432 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:37 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:37.108+0000 7f7ae777a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T03:23:37.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:37 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:37.431+0000 7f7ae777a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T03:23:38.800 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:38.525+0000 7f7ae777a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T03:23:39.410 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:39.268+0000 7f7ae777a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T03:23:39.711 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:39.410+0000 7f7ae777a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T03:23:40.049 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:23:39.710+0000 7f7ae777a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T03:23:40.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: Active manager daemon x restarted 2024-04-13T03:23:40.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: Activating manager daemon x 2024-04-13T03:23:40.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: osdmap e3: 0 total, 0 up, 0 in 2024-04-13T03:23:40.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: mgrmap e10: x(active, starting, since 0.00992947s) 2024-04-13T03:23:40.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:23:40.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-13T03:23:40.049 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-13T03:23:40.050 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-13T03:23:40.050 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-13T03:23:40.050 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:39 smithi016 ceph-mon[31772]: Manager daemon x is now available 2024-04-13T03:23:41.061 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:41.061 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-13T03:23:41.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-13T03:23:41.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:40 smithi016 ceph-mon[31772]: [13/Apr/2024:03:23:40] ENGINE Bus STARTING 2024-04-13T03:23:41.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:40 smithi016 ceph-mon[31772]: mgrmap e11: x(active, since 1.01072s) 2024-04-13T03:23:41.408 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-04-13T03:23:41.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-13T03:23:41.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-13T03:23:41.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-04-13T03:23:41.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.516+0000 7f050ec00700 1 Processor -- start 2024-04-13T03:23:41.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.517+0000 7f050ec00700 1 -- start start 2024-04-13T03:23:41.409 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.517+0000 7f050ec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f0508107970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:41.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.517+0000 7f050ec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0508107f40 con 0x7f0508107550 2024-04-13T03:23:41.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.518+0000 7f0507e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f0508107970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:41.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.518+0000 7f0507e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f0508107970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51836/0 (socket says 172.21.15.16:51836) 2024-04-13T03:23:41.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.518+0000 7f0507e00700 1 -- 172.21.15.16:0/4004533799 learned_addr learned my addr 172.21.15.16:0/4004533799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:41.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.518+0000 7f0507e00700 1 -- 172.21.15.16:0/4004533799 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0508108750 con 0x7f0508107550 2024-04-13T03:23:41.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.518+0000 7f0507e00700 1 --2- 172.21.15.16:0/4004533799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f0508107970 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009e80 tx=0x7f04f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913ba020fe2f8d0e server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:41.413 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.519+0000 7f0507400700 1 -- 172.21.15.16:0/4004533799 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f8017070 con 0x7f0508107550 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.519+0000 7f0507400700 1 -- 172.21.15.16:0/4004533799 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f800b8c0 con 0x7f0508107550 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.519+0000 7f0507400700 1 -- 172.21.15.16:0/4004533799 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f80045c0 con 0x7f0508107550 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.519+0000 7f050ec00700 1 -- 172.21.15.16:0/4004533799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 msgr2=0x7f0508107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.519+0000 7f050ec00700 1 --2- 172.21.15.16:0/4004533799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f0508107970 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009e80 tx=0x7f04f800b330 comp rx=0 tx=0).stop 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.520+0000 7f050ec00700 1 -- 172.21.15.16:0/4004533799 shutdown_connections 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.520+0000 7f050ec00700 1 --2- 172.21.15.16:0/4004533799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f0508107970 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.520+0000 7f050ec00700 1 -- 172.21.15.16:0/4004533799 >> 172.21.15.16:0/4004533799 conn(0x7f0508102b50 msgr2=0x7f0508104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.520+0000 7f050ec00700 1 -- 172.21.15.16:0/4004533799 shutdown_connections 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.520+0000 7f050ec00700 1 -- 172.21.15.16:0/4004533799 wait complete. 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.521+0000 7f050ec00700 1 Processor -- start 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.521+0000 7f050ec00700 1 -- start start 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.521+0000 7f050ec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f050819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:41.414 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.521+0000 7f050ec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f050810bb00 con 0x7f0508107550 2024-04-13T03:23:41.415 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.522+0000 7f0507e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f050819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:41.415 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.522+0000 7f0507e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f050819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51840/0 (socket says 172.21.15.16:51840) 2024-04-13T03:23:41.415 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.522+0000 7f0507e00700 1 -- 172.21.15.16:0/2458993704 learned_addr learned my addr 172.21.15.16:0/2458993704 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:41.415 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.522+0000 7f0507e00700 1 -- 172.21.15.16:0/2458993704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050819cc30 con 0x7f0508107550 2024-04-13T03:23:41.415 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.522+0000 7f0507e00700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f050819c6f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009530 tx=0x7f04f8003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:41.415 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.523+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f8020020 con 0x7f0508107550 2024-04-13T03:23:41.415 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.523+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f800bba0 con 0x7f0508107550 2024-04-13T03:23:41.415 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.523+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f8017040 con 0x7f0508107550 2024-04-13T03:23:41.419 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.523+0000 7f050ec00700 1 -- 172.21.15.16:0/2458993704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050819cec0 con 0x7f0508107550 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.523+0000 7f050ec00700 1 -- 172.21.15.16:0/2458993704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f050819fb60 con 0x7f0508107550 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.524+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f04f8023020 con 0x7f0508107550 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.525+0000 7f0505600700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 0x7f04ec03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.525+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 --> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f04ec03c230 con 0x7f04ec039660 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.525+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f04f800f050 con 0x7f0508107550 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.525+0000 7f04ff400700 1 -- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 msgr2=0x7f04ec03bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/164620940 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.525+0000 7f04ff400700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 0x7f04ec03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.726+0000 7f04ff400700 1 -- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 msgr2=0x7f04ec03bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/164620940 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:33.726+0000 7f04ff400700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 0x7f04ec03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:34.127+0000 7f04ff400700 1 -- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 msgr2=0x7f04ec03bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/164620940 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:34.127+0000 7f04ff400700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 0x7f04ec03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:34.928+0000 7f04ff400700 1 -- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 msgr2=0x7f04ec03bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/164620940 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:34.928+0000 7f04ff400700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 0x7f04ec03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-13T03:23:41.420 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:36.530+0000 7f04ff400700 1 -- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 msgr2=0x7f04ec03bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/164620940 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:36.530+0000 7f04ff400700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 0x7f04ec03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:39.722+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f04f8012430 con 0x7f0508107550 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:39.722+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 msgr2=0x7f04ec03bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:39.722+0000 7f0505600700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 0x7f04ec03bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.722+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f04f804ee40 con 0x7f0508107550 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.723+0000 7f0505600700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f04ec03cff0 0x7f04ec03f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.723+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f04f8027e70 con 0x7f04ec03cff0 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.724+0000 7f04ff400700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f04ec03cff0 0x7f04ec03f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.725+0000 7f04ff400700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f04ec03cff0 0x7f04ec03f3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f04f0003a10 tx=0x7f04f0009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.726+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f04f8027e70 con 0x7f04ec03cff0 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.733+0000 7f050ec00700 1 -- 172.21.15.16:0/2458993704 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f0508064110 con 0x7f04ec03cff0 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.734+0000 7f0505600700 1 -- 172.21.15.16:0/2458993704 <== mgr.14152 v2:172.21.15.16:6800/1721562410 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f0508064110 con 0x7f04ec03cff0 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.734+0000 7f050ec00700 1 -- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f04ec03cff0 msgr2=0x7f04ec03f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.734+0000 7f050ec00700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f04ec03cff0 0x7f04ec03f3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f04f0003a10 tx=0x7f04f0009270 comp rx=0 tx=0).stop 2024-04-13T03:23:41.421 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.734+0000 7f050ec00700 1 -- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 msgr2=0x7f050819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.734+0000 7f050ec00700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f050819c6f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009530 tx=0x7f04f8003d00 comp rx=0 tx=0).stop 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.734+0000 7f050ec00700 1 -- 172.21.15.16:0/2458993704 shutdown_connections 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.735+0000 7f050ec00700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f04ec03cff0 0x7f04ec03f3e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.735+0000 7f050ec00700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0508107550 0x7f050819c6f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.735+0000 7f050ec00700 1 --2- 172.21.15.16:0/2458993704 >> [v2:172.21.15.16:6800/164620940,v1:172.21.15.16:6801/164620940] conn(0x7f04ec039660 0x7f04ec03bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.735+0000 7f050ec00700 1 -- 172.21.15.16:0/2458993704 >> 172.21.15.16:0/2458993704 conn(0x7f0508102b50 msgr2=0x7f0508103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.735+0000 7f050ec00700 1 -- 172.21.15.16:0/2458993704 shutdown_connections 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:40.735+0000 7f050ec00700 1 -- 172.21.15.16:0/2458993704 wait complete. 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-04-13T03:23:41.422 INFO:teuthology.orchestra.run.smithi016.stdout:Generating a dashboard self-signed certificate... 2024-04-13T03:23:42.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:41 smithi016 ceph-mon[31772]: [13/Apr/2024:03:23:40] ENGINE Serving on https://172.21.15.16:7150 2024-04-13T03:23:42.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:41 smithi016 ceph-mon[31772]: [13/Apr/2024:03:23:40] ENGINE Serving on http://172.21.15.16:8765 2024-04-13T03:23:42.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:41 smithi016 ceph-mon[31772]: [13/Apr/2024:03:23:40] ENGINE Bus STARTED 2024-04-13T03:23:42.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:41 smithi016 ceph-mon[31772]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-13T03:23:42.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:41 smithi016 ceph-mon[31772]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-13T03:23:42.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:42 smithi016 ceph-mon[31772]: mgrmap e12: x(active, since 2s) 2024-04-13T03:23:42.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:42 smithi016 ceph-mon[31772]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:42.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:42.949 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:43.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-13T03:23:43.304 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.030+0000 7f5e7bc00700 1 Processor -- start 2024-04-13T03:23:43.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.030+0000 7f5e7bc00700 1 -- start start 2024-04-13T03:23:43.305 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.031+0000 7f5e7bc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74108a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:43.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.031+0000 7f5e7bc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5e74109020 con 0x7f5e74108630 2024-04-13T03:23:43.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.031+0000 7f5e79200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74108a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:43.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.032+0000 7f5e79200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74108a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35320/0 (socket says 172.21.15.16:35320) 2024-04-13T03:23:43.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.032+0000 7f5e79200700 1 -- 172.21.15.16:0/3465452599 learned_addr learned my addr 172.21.15.16:0/3465452599 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:43.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.032+0000 7f5e79200700 1 -- 172.21.15.16:0/3465452599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e741097e0 con 0x7f5e74108630 2024-04-13T03:23:43.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.032+0000 7f5e79200700 1 --2- 172.21.15.16:0/3465452599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74108a50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5e640098a0 tx=0x7f5e64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921fcd05031d33e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:43.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.033+0000 7f5e73e00700 1 -- 172.21.15.16:0/3465452599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e64004160 con 0x7f5e74108630 2024-04-13T03:23:43.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.033+0000 7f5e73e00700 1 -- 172.21.15.16:0/3465452599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e64004300 con 0x7f5e74108630 2024-04-13T03:23:43.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.033+0000 7f5e73e00700 1 -- 172.21.15.16:0/3465452599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e64011b20 con 0x7f5e74108630 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.033+0000 7f5e7bc00700 1 -- 172.21.15.16:0/3465452599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 msgr2=0x7f5e74108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.033+0000 7f5e7bc00700 1 --2- 172.21.15.16:0/3465452599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74108a50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5e640098a0 tx=0x7f5e64009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.034+0000 7f5e7bc00700 1 -- 172.21.15.16:0/3465452599 shutdown_connections 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.034+0000 7f5e7bc00700 1 --2- 172.21.15.16:0/3465452599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74108a50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.034+0000 7f5e7bc00700 1 -- 172.21.15.16:0/3465452599 >> 172.21.15.16:0/3465452599 conn(0x7f5e74103c50 msgr2=0x7f5e74106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.034+0000 7f5e7bc00700 1 -- 172.21.15.16:0/3465452599 shutdown_connections 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.034+0000 7f5e7bc00700 1 -- 172.21.15.16:0/3465452599 wait complete. 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.035+0000 7f5e7bc00700 1 Processor -- start 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.035+0000 7f5e7bc00700 1 -- start start 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.035+0000 7f5e7bc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74195590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.035+0000 7f5e7bc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7410cb90 con 0x7f5e74108630 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.036+0000 7f5e79200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74195590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.036+0000 7f5e79200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74195590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35336/0 (socket says 172.21.15.16:35336) 2024-04-13T03:23:43.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.036+0000 7f5e79200700 1 -- 172.21.15.16:0/1184866419 learned_addr learned my addr 172.21.15.16:0/1184866419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.036+0000 7f5e79200700 1 -- 172.21.15.16:0/1184866419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e74195ad0 con 0x7f5e74108630 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.037+0000 7f5e79200700 1 --2- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74195590 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5e640117e0 tx=0x7f5e64011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.037+0000 7f5e72000700 1 -- 172.21.15.16:0/1184866419 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e64003bf0 con 0x7f5e74108630 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.037+0000 7f5e72000700 1 -- 172.21.15.16:0/1184866419 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e6401a910 con 0x7f5e74108630 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.037+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e74195d60 con 0x7f5e74108630 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.037+0000 7f5e72000700 1 -- 172.21.15.16:0/1184866419 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e64024970 con 0x7f5e74108630 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.037+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e741922a0 con 0x7f5e74108630 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.038+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e74195f30 con 0x7f5e74108630 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.038+0000 7f5e72000700 1 -- 172.21.15.16:0/1184866419 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f5e64022020 con 0x7f5e74108630 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.039+0000 7f5e72000700 1 --2- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5e58039660 0x7f5e5803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.039+0000 7f5e72000700 1 -- 172.21.15.16:0/1184866419 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5e6404dc60 con 0x7f5e74108630 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.039+0000 7f5e6be00700 1 --2- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5e58039660 0x7f5e5803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.040+0000 7f5e6be00700 1 --2- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5e58039660 0x7f5e5803bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c006fd0 tx=0x7f5e5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:43.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.045+0000 7f5e72000700 1 -- 172.21.15.16:0/1184866419 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e74195f30 con 0x7f5e74108630 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.285+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f5e741928a0 con 0x7f5e58039660 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.451+0000 7f5e72000700 1 -- 172.21.15.16:0/1184866419 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f5e741928a0 con 0x7f5e58039660 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5e58039660 msgr2=0x7f5e5803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 --2- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5e58039660 0x7f5e5803bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c006fd0 tx=0x7f5e5c006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 msgr2=0x7f5e74195590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 --2- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74195590 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5e640117e0 tx=0x7f5e64011ba0 comp rx=0 tx=0).stop 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 shutdown_connections 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 --2- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5e58039660 0x7f5e5803bb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 --2- 172.21.15.16:0/1184866419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e74108630 0x7f5e74195590 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 >> 172.21.15.16:0/1184866419 conn(0x7f5e74103c50 msgr2=0x7f5e74104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 shutdown_connections 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:42.456+0000 7f5e7bc00700 1 -- 172.21.15.16:0/1184866419 wait complete. 2024-04-13T03:23:43.314 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-04-13T03:23:43.928 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:43 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:45.190 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:44 smithi016 ceph-mon[31772]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:45.190 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$mGzeebdqJUeRtWUX6/lq8OZntiwjU8YcCDcPy0.315nc9QEaz0mG.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712978624, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.972+0000 7f97ca600700 1 Processor -- start 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.972+0000 7f97ca600700 1 -- start start 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.973+0000 7f97ca600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c4108a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.973+0000 7f97ca600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4109020 con 0x7f97c4108630 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.973+0000 7f97c3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c4108a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.974+0000 7f97c3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c4108a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35348/0 (socket says 172.21.15.16:35348) 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.974+0000 7f97c3400700 1 -- 172.21.15.16:0/1948598631 learned_addr learned my addr 172.21.15.16:0/1948598631 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.974+0000 7f97c3400700 1 -- 172.21.15.16:0/1948598631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c41097e0 con 0x7f97c4108630 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.975+0000 7f97c3400700 1 --2- 172.21.15.16:0/1948598631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c4108a50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f97b40098a0 tx=0x7f97b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19f12959e819948 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.975+0000 7f97c2000700 1 -- 172.21.15.16:0/1948598631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97b4004160 con 0x7f97c4108630 2024-04-13T03:23:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.975+0000 7f97c2000700 1 -- 172.21.15.16:0/1948598631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b4004300 con 0x7f97c4108630 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.975+0000 7f97c2000700 1 -- 172.21.15.16:0/1948598631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97b4011b20 con 0x7f97c4108630 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.976+0000 7f97ca600700 1 -- 172.21.15.16:0/1948598631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 msgr2=0x7f97c4108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.976+0000 7f97ca600700 1 --2- 172.21.15.16:0/1948598631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c4108a50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f97b40098a0 tx=0x7f97b4009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.976+0000 7f97ca600700 1 -- 172.21.15.16:0/1948598631 shutdown_connections 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.976+0000 7f97ca600700 1 --2- 172.21.15.16:0/1948598631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c4108a50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.976+0000 7f97ca600700 1 -- 172.21.15.16:0/1948598631 >> 172.21.15.16:0/1948598631 conn(0x7f97c4103c50 msgr2=0x7f97c4106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.976+0000 7f97ca600700 1 -- 172.21.15.16:0/1948598631 shutdown_connections 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.976+0000 7f97ca600700 1 -- 172.21.15.16:0/1948598631 wait complete. 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.977+0000 7f97ca600700 1 Processor -- start 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.977+0000 7f97ca600700 1 -- start start 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.978+0000 7f97ca600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.978+0000 7f97ca600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c410cb90 con 0x7f97c4108630 2024-04-13T03:23:45.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.978+0000 7f97c3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.978+0000 7f97c3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c419e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35362/0 (socket says 172.21.15.16:35362) 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.978+0000 7f97c3400700 1 -- 172.21.15.16:0/1776143253 learned_addr learned my addr 172.21.15.16:0/1776143253 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.978+0000 7f97c3400700 1 -- 172.21.15.16:0/1776143253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c419ecc0 con 0x7f97c4108630 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.979+0000 7f97c3400700 1 --2- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c419e780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f97b40117e0 tx=0x7f97b4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.979+0000 7f97bbe00700 1 -- 172.21.15.16:0/1776143253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97b4003bf0 con 0x7f97c4108630 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.979+0000 7f97bbe00700 1 -- 172.21.15.16:0/1776143253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b401a910 con 0x7f97c4108630 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.979+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c419ef50 con 0x7f97c4108630 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.979+0000 7f97bbe00700 1 -- 172.21.15.16:0/1776143253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f97b4024970 con 0x7f97c4108630 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.979+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c419f3b0 con 0x7f97c4108630 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.981+0000 7f97bbe00700 1 -- 172.21.15.16:0/1776143253 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f97b4022020 con 0x7f97c4108630 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.981+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c4064110 con 0x7f97c4108630 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.981+0000 7f97bbe00700 1 --2- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f97a8039660 0x7f97a803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:45.202 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.981+0000 7f97bbe00700 1 -- 172.21.15.16:0/1776143253 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f97b404db60 con 0x7f97c4108630 2024-04-13T03:23:45.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.981+0000 7f97c2a00700 1 --2- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f97a8039660 0x7f97a803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:45.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.982+0000 7f97c2a00700 1 --2- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f97a8039660 0x7f97a803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f97b0006fd0 tx=0x7f97b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:45.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:43.988+0000 7f97bbe00700 1 -- 172.21.15.16:0/1776143253 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97b40212c0 con 0x7f97c4108630 2024-04-13T03:23:45.203 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.259+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f97c41a1d00 con 0x7f97a8039660 2024-04-13T03:23:45.204 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.512+0000 7f97bbe00700 1 -- 172.21.15.16:0/1776143253 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f97c41a1d00 con 0x7f97a8039660 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.514+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f97a8039660 msgr2=0x7f97a803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.514+0000 7f97ca600700 1 --2- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f97a8039660 0x7f97a803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f97b0006fd0 tx=0x7f97b0006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.514+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 msgr2=0x7f97c419e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.514+0000 7f97ca600700 1 --2- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c419e780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f97b40117e0 tx=0x7f97b4011ba0 comp rx=0 tx=0).stop 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.515+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 shutdown_connections 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.515+0000 7f97ca600700 1 --2- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f97a8039660 0x7f97a803bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.515+0000 7f97ca600700 1 --2- 172.21.15.16:0/1776143253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c4108630 0x7f97c419e780 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.515+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 >> 172.21.15.16:0/1776143253 conn(0x7f97c4103c50 msgr2=0x7f97c4104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.515+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 shutdown_connections 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:44.515+0000 7f97ca600700 1 -- 172.21.15.16:0/1776143253 wait complete. 2024-04-13T03:23:45.205 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-04-13T03:23:46.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:47.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:47 smithi016 ceph-mon[31772]: mgrmap e13: x(active, since 6s) 2024-04-13T03:23:47.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:47 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/12437243' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-13T03:23:47.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-04-13T03:23:47.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.830+0000 7f1b6c400700 1 Processor -- start 2024-04-13T03:23:47.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.830+0000 7f1b6c400700 1 -- start start 2024-04-13T03:23:47.616 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.830+0000 7f1b6c400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6407c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:47.616 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.830+0000 7f1b6c400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6407c7f0 con 0x7f1b6407de00 2024-04-13T03:23:47.616 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.831+0000 7f1b69a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6407c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:47.616 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.831+0000 7f1b69a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6407c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35368/0 (socket says 172.21.15.16:35368) 2024-04-13T03:23:47.616 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.831+0000 7f1b69a00700 1 -- 172.21.15.16:0/3473896898 learned_addr learned my addr 172.21.15.16:0/3473896898 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:47.616 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.831+0000 7f1b69a00700 1 -- 172.21.15.16:0/3473896898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6407c970 con 0x7f1b6407de00 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.832+0000 7f1b69a00700 1 --2- 172.21.15.16:0/3473896898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6407c2b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b580098a0 tx=0x7f1b58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52749e4182e62a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.832+0000 7f1b63e00700 1 -- 172.21.15.16:0/3473896898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b58004160 con 0x7f1b6407de00 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.832+0000 7f1b63e00700 1 -- 172.21.15.16:0/3473896898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b58004300 con 0x7f1b6407de00 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.832+0000 7f1b63e00700 1 -- 172.21.15.16:0/3473896898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b58011b20 con 0x7f1b6407de00 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.833+0000 7f1b6c400700 1 -- 172.21.15.16:0/3473896898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 msgr2=0x7f1b6407c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.833+0000 7f1b6c400700 1 --2- 172.21.15.16:0/3473896898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6407c2b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b580098a0 tx=0x7f1b58009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.833+0000 7f1b6c400700 1 -- 172.21.15.16:0/3473896898 shutdown_connections 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.833+0000 7f1b6c400700 1 --2- 172.21.15.16:0/3473896898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6407c2b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.833+0000 7f1b6c400700 1 -- 172.21.15.16:0/3473896898 >> 172.21.15.16:0/3473896898 conn(0x7f1b64103c50 msgr2=0x7f1b64106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:47.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.834+0000 7f1b6c400700 1 -- 172.21.15.16:0/3473896898 shutdown_connections 2024-04-13T03:23:47.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.834+0000 7f1b6c400700 1 -- 172.21.15.16:0/3473896898 wait complete. 2024-04-13T03:23:47.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.835+0000 7f1b6c400700 1 Processor -- start 2024-04-13T03:23:47.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.835+0000 7f1b6c400700 1 -- start start 2024-04-13T03:23:47.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.835+0000 7f1b6c400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:47.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.835+0000 7f1b6c400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6410ec10 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.836+0000 7f1b69a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.836+0000 7f1b69a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6419e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35382/0 (socket says 172.21.15.16:35382) 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.836+0000 7f1b69a00700 1 -- 172.21.15.16:0/12437243 learned_addr learned my addr 172.21.15.16:0/12437243 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.836+0000 7f1b69a00700 1 -- 172.21.15.16:0/12437243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6419eca0 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.837+0000 7f1b69a00700 1 --2- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6419e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b580117e0 tx=0x7f1b58011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.837+0000 7f1b62000700 1 -- 172.21.15.16:0/12437243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b58003bf0 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.837+0000 7f1b62000700 1 -- 172.21.15.16:0/12437243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b5801a910 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.837+0000 7f1b62000700 1 -- 172.21.15.16:0/12437243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b58024970 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.837+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6419ef30 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.837+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b6419f390 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.838+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b64064110 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.839+0000 7f1b62000700 1 -- 172.21.15.16:0/12437243 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f1b58022020 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.839+0000 7f1b62000700 1 --2- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1b4c039660 0x7f1b4c03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.839+0000 7f1b62000700 1 -- 172.21.15.16:0/12437243 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1b5804dbb0 con 0x7f1b6407de00 2024-04-13T03:23:47.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.840+0000 7f1b69000700 1 --2- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1b4c039660 0x7f1b4c03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.841+0000 7f1b69000700 1 --2- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1b4c039660 0x7f1b4c03bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1b50006fd0 tx=0x7f1b50006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:45.846+0000 7f1b62000700 1 -- 172.21.15.16:0/12437243 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b58011e20 con 0x7f1b6407de00 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.085+0000 7f1b62000700 1 -- 172.21.15.16:0/12437243 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1b5801a430 con 0x7f1b6407de00 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.113+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f1b641a1e20 con 0x7f1b6407de00 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.114+0000 7f1b62000700 1 -- 172.21.15.16:0/12437243 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f1b5800b3e0 con 0x7f1b6407de00 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1b4c039660 msgr2=0x7f1b4c03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 --2- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1b4c039660 0x7f1b4c03bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1b50006fd0 tx=0x7f1b50006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 msgr2=0x7f1b6419e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 --2- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6419e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b580117e0 tx=0x7f1b58011c10 comp rx=0 tx=0).stop 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 shutdown_connections 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 --2- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1b4c039660 0x7f1b4c03bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 --2- 172.21.15.16:0/12437243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b6407de00 0x7f1b6419e760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 >> 172.21.15.16:0/12437243 conn(0x7f1b64103c50 msgr2=0x7f1b64104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:47.624 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 shutdown_connections 2024-04-13T03:23:47.625 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:46.117+0000 7f1b6c400700 1 -- 172.21.15.16:0/12437243 wait complete. 2024-04-13T03:23:47.658 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-13T03:23:47.658 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-04-13T03:23:47.681 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-13T03:23:47.682 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-04-13T03:23:47.682 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-04-13T03:23:47.682 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-13T03:23:47.689 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-04-13T03:23:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016.front.sepia.ceph.com:8443/ 2024-04-13T03:23:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-04-13T03:23:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: Password: 35j79tc20b 2024-04-13T03:23:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:47.691 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config directory 2024-04-13T03:23:47.693 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling autotune for osd_memory_target 2024-04-13T03:23:49.261 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.321+0000 7ff9d9400700 1 Processor -- start 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.321+0000 7ff9d9400700 1 -- start start 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.321+0000 7ff9d9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d410b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.321+0000 7ff9d9400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d407bdc0 con 0x7ff9d41091c0 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.322+0000 7ff9d2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d410b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.322+0000 7ff9d2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d410b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35390/0 (socket says 172.21.15.16:35390) 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.322+0000 7ff9d2800700 1 -- 172.21.15.16:0/1773655079 learned_addr learned my addr 172.21.15.16:0/1773655079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.322+0000 7ff9d2800700 1 -- 172.21.15.16:0/1773655079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d410baf0 con 0x7ff9d41091c0 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.323+0000 7ff9d2800700 1 --2- 172.21.15.16:0/1773655079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d410b5b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40098a0 tx=0x7ff9c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5dc68f6fb89bcb server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.323+0000 7ff9d1400700 1 -- 172.21.15.16:0/1773655079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9c4004160 con 0x7ff9d41091c0 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.323+0000 7ff9d1400700 1 -- 172.21.15.16:0/1773655079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9c4004300 con 0x7ff9d41091c0 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.323+0000 7ff9d1400700 1 -- 172.21.15.16:0/1773655079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9c4011b20 con 0x7ff9d41091c0 2024-04-13T03:23:49.262 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.324+0000 7ff9d9400700 1 -- 172.21.15.16:0/1773655079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 msgr2=0x7ff9d410b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.324+0000 7ff9d9400700 1 --2- 172.21.15.16:0/1773655079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d410b5b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40098a0 tx=0x7ff9c4009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.324+0000 7ff9d9400700 1 -- 172.21.15.16:0/1773655079 shutdown_connections 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.324+0000 7ff9d9400700 1 --2- 172.21.15.16:0/1773655079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d410b5b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.324+0000 7ff9d9400700 1 -- 172.21.15.16:0/1773655079 >> 172.21.15.16:0/1773655079 conn(0x7ff9d4102c10 msgr2=0x7ff9d4105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.324+0000 7ff9d9400700 1 -- 172.21.15.16:0/1773655079 shutdown_connections 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.324+0000 7ff9d9400700 1 -- 172.21.15.16:0/1773655079 wait complete. 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.325+0000 7ff9d9400700 1 Processor -- start 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.325+0000 7ff9d9400700 1 -- start start 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.326+0000 7ff9d9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.326+0000 7ff9d9400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d410ca90 con 0x7ff9d41091c0 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.326+0000 7ff9d2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.326+0000 7ff9d2800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d419e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35396/0 (socket says 172.21.15.16:35396) 2024-04-13T03:23:49.263 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.326+0000 7ff9d2800700 1 -- 172.21.15.16:0/351835456 learned_addr learned my addr 172.21.15.16:0/351835456 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.326+0000 7ff9d2800700 1 -- 172.21.15.16:0/351835456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d419ed50 con 0x7ff9d41091c0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.327+0000 7ff9d2800700 1 --2- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d419e810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40116a0 tx=0x7ff9c4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.327+0000 7ff9cb400700 1 -- 172.21.15.16:0/351835456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9c4003a40 con 0x7ff9d41091c0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.327+0000 7ff9cb400700 1 -- 172.21.15.16:0/351835456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9c4003be0 con 0x7ff9d41091c0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.327+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d419efe0 con 0x7ff9d41091c0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.327+0000 7ff9cb400700 1 -- 172.21.15.16:0/351835456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9c401a750 con 0x7ff9d41091c0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.327+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d419f440 con 0x7ff9d41091c0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.329+0000 7ff9cb400700 1 -- 172.21.15.16:0/351835456 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff9c4024430 con 0x7ff9d41091c0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.329+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d4064110 con 0x7ff9d41091c0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.329+0000 7ff9cb400700 1 --2- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff9b80396b0 0x7ff9b803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.329+0000 7ff9cb400700 1 -- 172.21.15.16:0/351835456 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff9c404df60 con 0x7ff9d41091c0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.329+0000 7ff9d1e00700 1 --2- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff9b80396b0 0x7ff9b803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.330+0000 7ff9d1e00700 1 --2- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff9b80396b0 0x7ff9b803bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0006fd0 tx=0x7ff9c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:49.264 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.335+0000 7ff9cb400700 1 -- 172.21.15.16:0/351835456 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff9c4011e20 con 0x7ff9d41091c0 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.564+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7ff9d41a1e10 con 0x7ff9d41091c0 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.565+0000 7ff9cb400700 1 -- 172.21.15.16:0/351835456 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7ff9c402d3f0 con 0x7ff9d41091c0 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff9b80396b0 msgr2=0x7ff9b803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 --2- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff9b80396b0 0x7ff9b803bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0006fd0 tx=0x7ff9c0006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 msgr2=0x7ff9d419e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 --2- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d419e810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40116a0 tx=0x7ff9c4011ba0 comp rx=0 tx=0).stop 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 shutdown_connections 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 --2- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff9b80396b0 0x7ff9b803bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 --2- 172.21.15.16:0/351835456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9d41091c0 0x7ff9d419e810 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 >> 172.21.15.16:0/351835456 conn(0x7ff9d4102c10 msgr2=0x7ff9d4103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 shutdown_connections 2024-04-13T03:23:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:48.569+0000 7ff9d9400700 1 -- 172.21.15.16:0/351835456 wait complete. 2024-04-13T03:23:50.922 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.893+0000 7fb552800700 1 Processor -- start 2024-04-13T03:23:50.922 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.893+0000 7fb552800700 1 -- start start 2024-04-13T03:23:50.922 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.893+0000 7fb552800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c10ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:50.922 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.893+0000 7fb552800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c10b090 con 0x7fb54c10a730 2024-04-13T03:23:50.922 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.894+0000 7fb551400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c10ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:50.922 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.894+0000 7fb551400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c10ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35260/0 (socket says 172.21.15.16:35260) 2024-04-13T03:23:50.924 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.894+0000 7fb551400700 1 -- 172.21.15.16:0/4035749364 learned_addr learned my addr 172.21.15.16:0/4035749364 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:50.924 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.894+0000 7fb551400700 1 -- 172.21.15.16:0/4035749364 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c10b210 con 0x7fb54c10a730 2024-04-13T03:23:50.924 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.895+0000 7fb551400700 1 --2- 172.21.15.16:0/4035749364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c10ab50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb5400098a0 tx=0x7fb540009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b855774284be4afe server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:50.924 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.895+0000 7fb54be00700 1 -- 172.21.15.16:0/4035749364 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb540004160 con 0x7fb54c10a730 2024-04-13T03:23:50.924 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.895+0000 7fb54be00700 1 -- 172.21.15.16:0/4035749364 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb540004300 con 0x7fb54c10a730 2024-04-13T03:23:50.924 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.895+0000 7fb54be00700 1 -- 172.21.15.16:0/4035749364 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb540011b20 con 0x7fb54c10a730 2024-04-13T03:23:50.924 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.896+0000 7fb552800700 1 -- 172.21.15.16:0/4035749364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 msgr2=0x7fb54c10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:50.924 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.896+0000 7fb552800700 1 --2- 172.21.15.16:0/4035749364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c10ab50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb5400098a0 tx=0x7fb540009bb0 comp rx=0 tx=0).stop 2024-04-13T03:23:50.924 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.896+0000 7fb552800700 1 -- 172.21.15.16:0/4035749364 shutdown_connections 2024-04-13T03:23:50.925 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.896+0000 7fb552800700 1 --2- 172.21.15.16:0/4035749364 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c10ab50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.896+0000 7fb552800700 1 -- 172.21.15.16:0/4035749364 >> 172.21.15.16:0/4035749364 conn(0x7fb54c07de00 msgr2=0x7fb54c1084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.896+0000 7fb552800700 1 -- 172.21.15.16:0/4035749364 shutdown_connections 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.896+0000 7fb552800700 1 -- 172.21.15.16:0/4035749364 wait complete. 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.897+0000 7fb552800700 1 Processor -- start 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.897+0000 7fb552800700 1 -- start start 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.898+0000 7fb552800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.898+0000 7fb552800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c10eba0 con 0x7fb54c10a730 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.898+0000 7fb551400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.898+0000 7fb551400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c19e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35268/0 (socket says 172.21.15.16:35268) 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.898+0000 7fb551400700 1 -- 172.21.15.16:0/3041602576 learned_addr learned my addr 172.21.15.16:0/3041602576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.898+0000 7fb551400700 1 -- 172.21.15.16:0/3041602576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c19eb50 con 0x7fb54c10a730 2024-04-13T03:23:50.928 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.899+0000 7fb551400700 1 --2- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c19e610 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb5400117e0 tx=0x7fb540011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:50.928 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.899+0000 7fb54a000700 1 -- 172.21.15.16:0/3041602576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb540003ba0 con 0x7fb54c10a730 2024-04-13T03:23:50.928 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.899+0000 7fb54a000700 1 -- 172.21.15.16:0/3041602576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54001a910 con 0x7fb54c10a730 2024-04-13T03:23:50.928 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.899+0000 7fb54a000700 1 -- 172.21.15.16:0/3041602576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5400248f0 con 0x7fb54c10a730 2024-04-13T03:23:50.928 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.899+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54c19ede0 con 0x7fb54c10a730 2024-04-13T03:23:50.928 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.899+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54c19f240 con 0x7fb54c10a730 2024-04-13T03:23:50.928 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.901+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54c064110 con 0x7fb54c10a730 2024-04-13T03:23:50.928 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.901+0000 7fb54a000700 1 -- 172.21.15.16:0/3041602576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb540022020 con 0x7fb54c10a730 2024-04-13T03:23:50.928 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.901+0000 7fb54a000700 1 --2- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb534039700 0x7fb53403bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:50.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.901+0000 7fb54a000700 1 -- 172.21.15.16:0/3041602576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb54004dd40 con 0x7fb54c10a730 2024-04-13T03:23:50.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.902+0000 7fb550a00700 1 --2- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb534039700 0x7fb53403bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:50.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.902+0000 7fb550a00700 1 --2- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb534039700 0x7fb53403bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb538006fd0 tx=0x7fb538006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:50.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:49.907+0000 7fb54a000700 1 -- 172.21.15.16:0/3041602576 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb540054080 con 0x7fb54c10a730 2024-04-13T03:23:50.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.254+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fb54c1a1bc0 con 0x7fb54c10a730 2024-04-13T03:23:50.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.257+0000 7fb54a000700 1 -- 172.21.15.16:0/3041602576 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fb54001ab70 con 0x7fb54c10a730 2024-04-13T03:23:50.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.259+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb534039700 msgr2=0x7fb53403bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:50.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.259+0000 7fb552800700 1 --2- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb534039700 0x7fb53403bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb538006fd0 tx=0x7fb538006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:50.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.259+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 msgr2=0x7fb54c19e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.259+0000 7fb552800700 1 --2- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c19e610 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb5400117e0 tx=0x7fb540011ba0 comp rx=0 tx=0).stop 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.260+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 shutdown_connections 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.260+0000 7fb552800700 1 --2- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb534039700 0x7fb53403bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.260+0000 7fb552800700 1 --2- 172.21.15.16:0/3041602576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c10a730 0x7fb54c19e610 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.260+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 >> 172.21.15.16:0/3041602576 conn(0x7fb54c07de00 msgr2=0x7fb54c107ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.260+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 shutdown_connections 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-04-13T03:23:50.260+0000 7fb552800700 1 -- 172.21.15.16:0/3041602576 wait complete. 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-13T03:23:50.931 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:50.932 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-13T03:23:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-04-13T03:23:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:50.932 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-04-13T03:23:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-13T03:23:50.932 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:23:50.932 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-04-13T03:23:51.058 INFO:tasks.cephadm:Fetching config... 2024-04-13T03:23:51.059 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:23:51.059 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-13T03:23:51.114 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-13T03:23:51.115 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:23:51.115 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-13T03:23:51.160 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-13T03:23:51.160 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:23:51.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-13T03:23:51.290 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-13T03:23:51.290 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:23:51.291 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-13T03:23:51.342 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-13T03:23:51.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwhJeSSvHu9dhxbS/akIjigavlveQN49JXAX2H978JTMWop1/YYTpw4pbcGU6uOmtMsJZ1rz2ivC7XBOIzwfH90HrvRGJ6fM71EuA9CuQzJHer3XX51egPnC4eFb15bsMQIy8VBANlGfUTyaTAM2g3NpXdw7yBdDvDfxmeSdfsyIWzf7+I1uSTIBIZYYbG9p930pchOcnW9uBIf0vke2kN4K08s8uJa0eklcW3Vdazh+3g4kDD+v3MvGZDYFyCv86OadLRd6mIYrfDYPCHkaweyanK75PHVMpuorHZ9LSyYEBMO4RUyEn9OcbizDzBE8Qw762+OkdqI9oJPB/gFCMdI9lyhByIoYX8smteLc6fY1oP92fXaZRglsTRIy1LXuNo9su41PTVAry10IwUZpNkx0OZYxnFb8vCEmxZg7xCamxMEixcwyfDba01Oc602QbSDm0R2n1T6hxjM00fpeG+ZsWCaJG9I7xeaAo3biikwV9aDGuwHhOYE/AHaYFt1+U= ceph-e6bf0314-f944-11ee-bc8d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-13T03:23:51.481 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwhJeSSvHu9dhxbS/akIjigavlveQN49JXAX2H978JTMWop1/YYTpw4pbcGU6uOmtMsJZ1rz2ivC7XBOIzwfH90HrvRGJ6fM71EuA9CuQzJHer3XX51egPnC4eFb15bsMQIy8VBANlGfUTyaTAM2g3NpXdw7yBdDvDfxmeSdfsyIWzf7+I1uSTIBIZYYbG9p930pchOcnW9uBIf0vke2kN4K08s8uJa0eklcW3Vdazh+3g4kDD+v3MvGZDYFyCv86OadLRd6mIYrfDYPCHkaweyanK75PHVMpuorHZ9LSyYEBMO4RUyEn9OcbizDzBE8Qw762+OkdqI9oJPB/gFCMdI9lyhByIoYX8smteLc6fY1oP92fXaZRglsTRIy1LXuNo9su41PTVAry10IwUZpNkx0OZYxnFb8vCEmxZg7xCamxMEixcwyfDba01Oc602QbSDm0R2n1T6hxjM00fpeG+ZsWCaJG9I7xeaAo3biikwV9aDGuwHhOYE/AHaYFt1+U= ceph-e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:23:51.519 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:51 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3041602576' entity='client.admin' 2024-04-13T03:23:51.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwhJeSSvHu9dhxbS/akIjigavlveQN49JXAX2H978JTMWop1/YYTpw4pbcGU6uOmtMsJZ1rz2ivC7XBOIzwfH90HrvRGJ6fM71EuA9CuQzJHer3XX51egPnC4eFb15bsMQIy8VBANlGfUTyaTAM2g3NpXdw7yBdDvDfxmeSdfsyIWzf7+I1uSTIBIZYYbG9p930pchOcnW9uBIf0vke2kN4K08s8uJa0eklcW3Vdazh+3g4kDD+v3MvGZDYFyCv86OadLRd6mIYrfDYPCHkaweyanK75PHVMpuorHZ9LSyYEBMO4RUyEn9OcbizDzBE8Qw762+OkdqI9oJPB/gFCMdI9lyhByIoYX8smteLc6fY1oP92fXaZRglsTRIy1LXuNo9su41PTVAry10IwUZpNkx0OZYxnFb8vCEmxZg7xCamxMEixcwyfDba01Oc602QbSDm0R2n1T6hxjM00fpeG+ZsWCaJG9I7xeaAo3biikwV9aDGuwHhOYE/AHaYFt1+U= ceph-e6bf0314-f944-11ee-bc8d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-13T03:23:51.589 INFO:teuthology.orchestra.run.smithi119.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwhJeSSvHu9dhxbS/akIjigavlveQN49JXAX2H978JTMWop1/YYTpw4pbcGU6uOmtMsJZ1rz2ivC7XBOIzwfH90HrvRGJ6fM71EuA9CuQzJHer3XX51egPnC4eFb15bsMQIy8VBANlGfUTyaTAM2g3NpXdw7yBdDvDfxmeSdfsyIWzf7+I1uSTIBIZYYbG9p930pchOcnW9uBIf0vke2kN4K08s8uJa0eklcW3Vdazh+3g4kDD+v3MvGZDYFyCv86OadLRd6mIYrfDYPCHkaweyanK75PHVMpuorHZ9LSyYEBMO4RUyEn9OcbizDzBE8Qw762+OkdqI9oJPB/gFCMdI9lyhByIoYX8smteLc6fY1oP92fXaZRglsTRIy1LXuNo9su41PTVAry10IwUZpNkx0OZYxnFb8vCEmxZg7xCamxMEixcwyfDba01Oc602QbSDm0R2n1T6hxjM00fpeG+ZsWCaJG9I7xeaAo3biikwV9aDGuwHhOYE/AHaYFt1+U= ceph-e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:23:51.604 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwhJeSSvHu9dhxbS/akIjigavlveQN49JXAX2H978JTMWop1/YYTpw4pbcGU6uOmtMsJZ1rz2ivC7XBOIzwfH90HrvRGJ6fM71EuA9CuQzJHer3XX51egPnC4eFb15bsMQIy8VBANlGfUTyaTAM2g3NpXdw7yBdDvDfxmeSdfsyIWzf7+I1uSTIBIZYYbG9p930pchOcnW9uBIf0vke2kN4K08s8uJa0eklcW3Vdazh+3g4kDD+v3MvGZDYFyCv86OadLRd6mIYrfDYPCHkaweyanK75PHVMpuorHZ9LSyYEBMO4RUyEn9OcbizDzBE8Qw762+OkdqI9oJPB/gFCMdI9lyhByIoYX8smteLc6fY1oP92fXaZRglsTRIy1LXuNo9su41PTVAry10IwUZpNkx0OZYxnFb8vCEmxZg7xCamxMEixcwyfDba01Oc602QbSDm0R2n1T6hxjM00fpeG+ZsWCaJG9I7xeaAo3biikwV9aDGuwHhOYE/AHaYFt1+U= ceph-e6bf0314-f944-11ee-bc8d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-13T03:23:51.651 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwhJeSSvHu9dhxbS/akIjigavlveQN49JXAX2H978JTMWop1/YYTpw4pbcGU6uOmtMsJZ1rz2ivC7XBOIzwfH90HrvRGJ6fM71EuA9CuQzJHer3XX51egPnC4eFb15bsMQIy8VBANlGfUTyaTAM2g3NpXdw7yBdDvDfxmeSdfsyIWzf7+I1uSTIBIZYYbG9p930pchOcnW9uBIf0vke2kN4K08s8uJa0eklcW3Vdazh+3g4kDD+v3MvGZDYFyCv86OadLRd6mIYrfDYPCHkaweyanK75PHVMpuorHZ9LSyYEBMO4RUyEn9OcbizDzBE8Qw762+OkdqI9oJPB/gFCMdI9lyhByIoYX8smteLc6fY1oP92fXaZRglsTRIy1LXuNo9su41PTVAry10IwUZpNkx0OZYxnFb8vCEmxZg7xCamxMEixcwyfDba01Oc602QbSDm0R2n1T6hxjM00fpeG+ZsWCaJG9I7xeaAo3biikwV9aDGuwHhOYE/AHaYFt1+U= ceph-e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:23:51.666 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-13T03:23:52.111 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:23:54.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.705+0000 7f228a600700 1 -- 172.21.15.16:0/411830685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 msgr2=0x7f22841090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.705+0000 7f228a600700 1 --2- 172.21.15.16:0/411830685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 0x7f22841090e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2274009910 tx=0x7f2274009c20 comp rx=0 tx=0).stop 2024-04-13T03:23:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.705+0000 7f228a600700 1 -- 172.21.15.16:0/411830685 shutdown_connections 2024-04-13T03:23:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.705+0000 7f228a600700 1 --2- 172.21.15.16:0/411830685 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 0x7f22841090e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.705+0000 7f228a600700 1 -- 172.21.15.16:0/411830685 >> 172.21.15.16:0/411830685 conn(0x7f2284077ba0 msgr2=0x7f228407a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.705+0000 7f228a600700 1 -- 172.21.15.16:0/411830685 shutdown_connections 2024-04-13T03:23:54.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.705+0000 7f228a600700 1 -- 172.21.15.16:0/411830685 wait complete. 2024-04-13T03:23:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.706+0000 7f228a600700 1 Processor -- start 2024-04-13T03:23:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.706+0000 7f228a600700 1 -- start start 2024-04-13T03:23:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.707+0000 7f228a600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 0x7f228419e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.707+0000 7f228a600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f228410ebe0 con 0x7f2284108cc0 2024-04-13T03:23:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.707+0000 7f2283400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 0x7f228419e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.707+0000 7f2283400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 0x7f228419e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35296/0 (socket says 172.21.15.16:35296) 2024-04-13T03:23:54.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.707+0000 7f2283400700 1 -- 172.21.15.16:0/3857404714 learned_addr learned my addr 172.21.15.16:0/3857404714 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:54.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.708+0000 7f2283400700 1 -- 172.21.15.16:0/3857404714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228419e790 con 0x7f2284108cc0 2024-04-13T03:23:54.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.708+0000 7f2283400700 1 --2- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 0x7f228419e250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2274006010 tx=0x7f2274005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:54.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.708+0000 7f227be00700 1 -- 172.21.15.16:0/3857404714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2274027070 con 0x7f2284108cc0 2024-04-13T03:23:54.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.708+0000 7f227be00700 1 -- 172.21.15.16:0/3857404714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f227400be40 con 0x7f2284108cc0 2024-04-13T03:23:54.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.708+0000 7f227be00700 1 -- 172.21.15.16:0/3857404714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f227400f460 con 0x7f2284108cc0 2024-04-13T03:23:54.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.708+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228419ea20 con 0x7f2284108cc0 2024-04-13T03:23:54.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.708+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22841a1710 con 0x7f2284108cc0 2024-04-13T03:23:54.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.710+0000 7f227be00700 1 -- 172.21.15.16:0/3857404714 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f227402a070 con 0x7f2284108cc0 2024-04-13T03:23:54.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.710+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f228410f360 con 0x7f2284108cc0 2024-04-13T03:23:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.710+0000 7f227be00700 1 --2- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2268039650 0x7f226803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.710+0000 7f227be00700 1 -- 172.21.15.16:0/3857404714 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f227401e030 con 0x7f2284108cc0 2024-04-13T03:23:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.710+0000 7f2282a00700 1 --2- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2268039650 0x7f226803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.711+0000 7f2282a00700 1 --2- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2268039650 0x7f226803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2270006fd0 tx=0x7f2270006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.716+0000 7f227be00700 1 -- 172.21.15.16:0/3857404714 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2274013070 con 0x7f2284108cc0 2024-04-13T03:23:54.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.951+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f228402e0b0 con 0x7f2284108cc0 2024-04-13T03:23:54.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.955+0000 7f227be00700 1 -- 172.21.15.16:0/3857404714 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f2274031020 con 0x7f2284108cc0 2024-04-13T03:23:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.959+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2268039650 msgr2=0x7f226803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:54.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.959+0000 7f228a600700 1 --2- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2268039650 0x7f226803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2270006fd0 tx=0x7f2270006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.959+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 msgr2=0x7f228419e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.959+0000 7f228a600700 1 --2- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 0x7f228419e250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2274006010 tx=0x7f2274005050 comp rx=0 tx=0).stop 2024-04-13T03:23:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.959+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 shutdown_connections 2024-04-13T03:23:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.959+0000 7f228a600700 1 --2- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2268039650 0x7f226803bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.959+0000 7f228a600700 1 --2- 172.21.15.16:0/3857404714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2284108cc0 0x7f228419e250 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.959+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 >> 172.21.15.16:0/3857404714 conn(0x7f2284077ba0 msgr2=0x7f2284078690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.960+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 shutdown_connections 2024-04-13T03:23:54.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:54.960+0000 7f228a600700 1 -- 172.21.15.16:0/3857404714 wait complete. 2024-04-13T03:23:55.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:55.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:55.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:23:55.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:55.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:55.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:55.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:55 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3857404714' entity='client.admin' 2024-04-13T03:23:55.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:55.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:55.709 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-13T03:23:55.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-13T03:23:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:23:58.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.041+0000 7fde98e00700 1 -- 172.21.15.16:0/2390227931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 msgr2=0x7fde94104b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:58.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.041+0000 7fde98e00700 1 --2- 172.21.15.16:0/2390227931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 0x7fde94104b70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fde84009910 tx=0x7fde84009c20 comp rx=0 tx=0).stop 2024-04-13T03:23:58.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.042+0000 7fde98e00700 1 -- 172.21.15.16:0/2390227931 shutdown_connections 2024-04-13T03:23:58.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.042+0000 7fde98e00700 1 --2- 172.21.15.16:0/2390227931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 0x7fde94104b70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:58.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.042+0000 7fde98e00700 1 -- 172.21.15.16:0/2390227931 >> 172.21.15.16:0/2390227931 conn(0x7fde940ffd30 msgr2=0x7fde94102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:58.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.042+0000 7fde98e00700 1 -- 172.21.15.16:0/2390227931 shutdown_connections 2024-04-13T03:23:58.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.042+0000 7fde98e00700 1 -- 172.21.15.16:0/2390227931 wait complete. 2024-04-13T03:23:58.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.043+0000 7fde98e00700 1 Processor -- start 2024-04-13T03:23:58.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.043+0000 7fde98e00700 1 -- start start 2024-04-13T03:23:58.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.044+0000 7fde98e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 0x7fde94199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:58.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.044+0000 7fde98e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde9410a7e0 con 0x7fde94104750 2024-04-13T03:23:58.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.044+0000 7fde91e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 0x7fde94199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:58.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.044+0000 7fde91e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 0x7fde94199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35324/0 (socket says 172.21.15.16:35324) 2024-04-13T03:23:58.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.044+0000 7fde91e00700 1 -- 172.21.15.16:0/2886965654 learned_addr learned my addr 172.21.15.16:0/2886965654 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:23:58.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.044+0000 7fde91e00700 1 -- 172.21.15.16:0/2886965654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9419a360 con 0x7fde94104750 2024-04-13T03:23:58.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.045+0000 7fde91e00700 1 --2- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 0x7fde94199e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde84006010 tx=0x7fde84005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:58.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.045+0000 7fde8aa00700 1 -- 172.21.15.16:0/2886965654 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde84027070 con 0x7fde94104750 2024-04-13T03:23:58.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.045+0000 7fde8aa00700 1 -- 172.21.15.16:0/2886965654 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde8400be40 con 0x7fde94104750 2024-04-13T03:23:58.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.045+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9419a5f0 con 0x7fde94104750 2024-04-13T03:23:58.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.045+0000 7fde8aa00700 1 -- 172.21.15.16:0/2886965654 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde8400f460 con 0x7fde94104750 2024-04-13T03:23:58.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.045+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9419d2e0 con 0x7fde94104750 2024-04-13T03:23:58.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.047+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde9410af60 con 0x7fde94104750 2024-04-13T03:23:58.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.047+0000 7fde8aa00700 1 -- 172.21.15.16:0/2886965654 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fde8402a070 con 0x7fde94104750 2024-04-13T03:23:58.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.048+0000 7fde8aa00700 1 --2- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fde78039650 0x7fde7803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:23:58.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.048+0000 7fde8aa00700 1 -- 172.21.15.16:0/2886965654 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fde8401e030 con 0x7fde94104750 2024-04-13T03:23:58.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.048+0000 7fde91400700 1 --2- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fde78039650 0x7fde7803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:23:58.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.049+0000 7fde91400700 1 --2- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fde78039650 0x7fde7803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fde80006fd0 tx=0x7fde80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:23:58.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.053+0000 7fde8aa00700 1 -- 172.21.15.16:0/2886965654 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde84031070 con 0x7fde94104750 2024-04-13T03:23:58.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.290+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fde9402e0b0 con 0x7fde78039650 2024-04-13T03:23:58.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.299+0000 7fde8aa00700 1 -- 172.21.15.16:0/2886965654 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fde9402e0b0 con 0x7fde78039650 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.302+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fde78039650 msgr2=0x7fde7803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.302+0000 7fde98e00700 1 --2- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fde78039650 0x7fde7803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fde80006fd0 tx=0x7fde80006e40 comp rx=0 tx=0).stop 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.302+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 msgr2=0x7fde94199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.302+0000 7fde98e00700 1 --2- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 0x7fde94199e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fde84006010 tx=0x7fde84005050 comp rx=0 tx=0).stop 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.302+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 shutdown_connections 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.302+0000 7fde98e00700 1 --2- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fde78039650 0x7fde7803bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.302+0000 7fde98e00700 1 --2- 172.21.15.16:0/2886965654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde94104750 0x7fde94199e20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.302+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 >> 172.21.15.16:0/2886965654 conn(0x7fde940ffd30 msgr2=0x7fde94100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.303+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 shutdown_connections 2024-04-13T03:23:58.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:23:58.303+0000 7fde98e00700 1 -- 172.21.15.16:0/2886965654 wait complete. 2024-04-13T03:23:59.112 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi119 2024-04-13T03:23:59.113 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:23:59.113 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.conf 2024-04-13T03:23:59.130 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:23:59.131 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:23:59.191 INFO:tasks.cephadm:Adding host smithi119 to orchestrator... 2024-04-13T03:23:59.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch host add smithi119 2024-04-13T03:23:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:59 smithi016 ceph-mon[31772]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:23:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:23:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:23:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:23:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:23:59 smithi016 ceph-mon[31772]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-13T03:23:59.620 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:24:00.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:00 smithi016 ceph-mon[31772]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:01.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:01 smithi016 ceph-mon[31772]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:01.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:01 smithi016 ceph-mon[31772]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:01.511 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:01 smithi016 ceph-mon[31772]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:24:01.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:01.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:01.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.684+0000 7f1da6a00700 1 -- 172.21.15.16:0/158041906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 msgr2=0x7f1da010aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.684+0000 7f1da6a00700 1 --2- 172.21.15.16:0/158041906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 0x7f1da010aac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d90009910 tx=0x7f1d90009c20 comp rx=0 tx=0).stop 2024-04-13T03:24:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.684+0000 7f1da6a00700 1 -- 172.21.15.16:0/158041906 shutdown_connections 2024-04-13T03:24:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.684+0000 7f1da6a00700 1 --2- 172.21.15.16:0/158041906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 0x7f1da010aac0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.684+0000 7f1da6a00700 1 -- 172.21.15.16:0/158041906 >> 172.21.15.16:0/158041906 conn(0x7f1da01001d0 msgr2=0x7f1da01025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.684+0000 7f1da6a00700 1 -- 172.21.15.16:0/158041906 shutdown_connections 2024-04-13T03:24:01.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.684+0000 7f1da6a00700 1 -- 172.21.15.16:0/158041906 wait complete. 2024-04-13T03:24:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.686+0000 7f1da6a00700 1 Processor -- start 2024-04-13T03:24:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.686+0000 7f1da6a00700 1 -- start start 2024-04-13T03:24:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.686+0000 7f1da6a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 0x7f1da019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:01.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.686+0000 7f1da6a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1da010ed20 con 0x7f1da010a6e0 2024-04-13T03:24:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.687+0000 7f1d9fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 0x7f1da019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.687+0000 7f1d9fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 0x7f1da019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33818/0 (socket says 172.21.15.16:33818) 2024-04-13T03:24:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.687+0000 7f1d9fe00700 1 -- 172.21.15.16:0/1176481380 learned_addr learned my addr 172.21.15.16:0/1176481380 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:24:01.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.687+0000 7f1d9fe00700 1 -- 172.21.15.16:0/1176481380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da019a740 con 0x7f1da010a6e0 2024-04-13T03:24:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.688+0000 7f1d9fe00700 1 --2- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 0x7f1da019a200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1d90006010 tx=0x7f1d90005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.688+0000 7f1d9cc00700 1 -- 172.21.15.16:0/1176481380 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d90026070 con 0x7f1da010a6e0 2024-04-13T03:24:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.688+0000 7f1d9cc00700 1 -- 172.21.15.16:0/1176481380 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d9000bbd0 con 0x7f1da010a6e0 2024-04-13T03:24:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.688+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da019aa30 con 0x7f1da010a6e0 2024-04-13T03:24:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.688+0000 7f1d9cc00700 1 -- 172.21.15.16:0/1176481380 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d90003910 con 0x7f1da010a6e0 2024-04-13T03:24:01.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.688+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da019ea80 con 0x7f1da010a6e0 2024-04-13T03:24:01.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.689+0000 7f1d9cc00700 1 -- 172.21.15.16:0/1176481380 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1d90029070 con 0x7f1da010a6e0 2024-04-13T03:24:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.689+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da0105e20 con 0x7f1da010a6e0 2024-04-13T03:24:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.690+0000 7f1d9cc00700 1 --2- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1d84039700 0x7f1d8403bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.690+0000 7f1d9cc00700 1 -- 172.21.15.16:0/1176481380 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1d9001e070 con 0x7f1da010a6e0 2024-04-13T03:24:01.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.690+0000 7f1d9f400700 1 --2- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1d84039700 0x7f1d8403bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:01.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.691+0000 7f1d9f400700 1 --2- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1d84039700 0x7f1d8403bbc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c006fd0 tx=0x7f1d8c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:01.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.696+0000 7f1d9cc00700 1 -- 172.21.15.16:0/1176481380 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1d90013070 con 0x7f1da010a6e0 2024-04-13T03:24:01.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:01.931+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}) -- 0x7f1da019b510 con 0x7f1d84039700 2024-04-13T03:24:03.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:03 smithi016 ceph-mon[31772]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:03.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:03 smithi016 ceph-mon[31772]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:04.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:04 smithi016 ceph-mon[31772]: Deploying cephadm binary to smithi119 2024-04-13T03:24:05.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:05 smithi016 ceph-mon[31772]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.964+0000 7f1d9cc00700 1 -- 172.21.15.16:0/1176481380 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f1da019b510 con 0x7f1d84039700 2024-04-13T03:24:06.965 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi119' with addr '172.21.15.119' 2024-04-13T03:24:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1d84039700 msgr2=0x7f1d8403bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 --2- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1d84039700 0x7f1d8403bbc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c006fd0 tx=0x7f1d8c006e40 comp rx=0 tx=0).stop 2024-04-13T03:24:06.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 msgr2=0x7f1da019a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 --2- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 0x7f1da019a200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1d90006010 tx=0x7f1d90005030 comp rx=0 tx=0).stop 2024-04-13T03:24:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 shutdown_connections 2024-04-13T03:24:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 --2- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1d84039700 0x7f1d8403bbc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 --2- 172.21.15.16:0/1176481380 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1da010a6e0 0x7f1da019a200 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 >> 172.21.15.16:0/1176481380 conn(0x7f1da01001d0 msgr2=0x7f1da010c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 shutdown_connections 2024-04-13T03:24:06.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:06.968+0000 7f1da6a00700 1 -- 172.21.15.16:0/1176481380 wait complete. 2024-04-13T03:24:07.547 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:07 smithi016 ceph-mon[31772]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:07.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:07.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:07.609 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch host ls --format=json 2024-04-13T03:24:08.058 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:24:08.476 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:08 smithi016 ceph-mon[31772]: Added host smithi119 2024-04-13T03:24:08.476 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:08.476 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:09.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:09 smithi016 ceph-mon[31772]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:09.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.882+0000 7f92bcc00700 1 -- 172.21.15.16:0/1009885607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 msgr2=0x7f92b8108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:09.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.882+0000 7f92bcc00700 1 --2- 172.21.15.16:0/1009885607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 0x7f92b8108920 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009910 tx=0x7f92a8009c20 comp rx=0 tx=0).stop 2024-04-13T03:24:09.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.883+0000 7f92bcc00700 1 -- 172.21.15.16:0/1009885607 shutdown_connections 2024-04-13T03:24:09.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.883+0000 7f92bcc00700 1 --2- 172.21.15.16:0/1009885607 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 0x7f92b8108920 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:09.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.883+0000 7f92bcc00700 1 -- 172.21.15.16:0/1009885607 >> 172.21.15.16:0/1009885607 conn(0x7f92b8076f10 msgr2=0x7f92b8077320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:09.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.883+0000 7f92bcc00700 1 -- 172.21.15.16:0/1009885607 shutdown_connections 2024-04-13T03:24:09.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.883+0000 7f92bcc00700 1 -- 172.21.15.16:0/1009885607 wait complete. 2024-04-13T03:24:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.884+0000 7f92bcc00700 1 Processor -- start 2024-04-13T03:24:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.884+0000 7f92bcc00700 1 -- start start 2024-04-13T03:24:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.884+0000 7f92bcc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 0x7f92b819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.884+0000 7f92bcc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f92b810ca80 con 0x7f92b8108540 2024-04-13T03:24:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.884+0000 7f92b5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 0x7f92b819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.885+0000 7f92b5e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 0x7f92b819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35874/0 (socket says 172.21.15.16:35874) 2024-04-13T03:24:09.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.885+0000 7f92b5e00700 1 -- 172.21.15.16:0/1887395875 learned_addr learned my addr 172.21.15.16:0/1887395875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:24:09.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.885+0000 7f92b5e00700 1 -- 172.21.15.16:0/1887395875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b819a6a0 con 0x7f92b8108540 2024-04-13T03:24:09.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.885+0000 7f92b5e00700 1 --2- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 0x7f92b819a160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f92a8006010 tx=0x7f92a8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:09.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.885+0000 7f92aea00700 1 -- 172.21.15.16:0/1887395875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92a8026070 con 0x7f92b8108540 2024-04-13T03:24:09.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.886+0000 7f92aea00700 1 -- 172.21.15.16:0/1887395875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a800bc30 con 0x7f92b8108540 2024-04-13T03:24:09.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.886+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b819a990 con 0x7f92b8108540 2024-04-13T03:24:09.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.886+0000 7f92aea00700 1 -- 172.21.15.16:0/1887395875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92a80039d0 con 0x7f92b8108540 2024-04-13T03:24:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.886+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b819ea30 con 0x7f92b8108540 2024-04-13T03:24:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.887+0000 7f92aea00700 1 -- 172.21.15.16:0/1887395875 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f92a8029070 con 0x7f92b8108540 2024-04-13T03:24:09.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.887+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92b8103c00 con 0x7f92b8108540 2024-04-13T03:24:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.887+0000 7f92aea00700 1 --2- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f929c0396b0 0x7f929c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.887+0000 7f92aea00700 1 -- 172.21.15.16:0/1887395875 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f92a801e070 con 0x7f92b8108540 2024-04-13T03:24:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.888+0000 7f92b5400700 1 --2- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f929c0396b0 0x7f929c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:09.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.888+0000 7f92b5400700 1 --2- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f929c0396b0 0x7f929c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f92a4006fd0 tx=0x7f92a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:09.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:09.893+0000 7f92aea00700 1 -- 172.21.15.16:0/1887395875 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f92a8013070 con 0x7f92b8108540 2024-04-13T03:24:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.128+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92b819b4c0 con 0x7f929c0396b0 2024-04-13T03:24:10.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.131+0000 7f92aea00700 1 -- 172.21.15.16:0/1887395875 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f92b819b4c0 con 0x7f929c0396b0 2024-04-13T03:24:10.133 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:24:10.133 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}] 2024-04-13T03:24:10.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.134+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f929c0396b0 msgr2=0x7f929c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:10.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.134+0000 7f92bcc00700 1 --2- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f929c0396b0 0x7f929c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f92a4006fd0 tx=0x7f92a4006e40 comp rx=0 tx=0).stop 2024-04-13T03:24:10.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.134+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 msgr2=0x7f92b819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:10.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.134+0000 7f92bcc00700 1 --2- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 0x7f92b819a160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f92a8006010 tx=0x7f92a8005030 comp rx=0 tx=0).stop 2024-04-13T03:24:10.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.135+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 shutdown_connections 2024-04-13T03:24:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.135+0000 7f92bcc00700 1 --2- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f929c0396b0 0x7f929c03bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.135+0000 7f92bcc00700 1 --2- 172.21.15.16:0/1887395875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92b8108540 0x7f92b819a160 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.135+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 >> 172.21.15.16:0/1887395875 conn(0x7f92b8076f10 msgr2=0x7f92b810ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.135+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 shutdown_connections 2024-04-13T03:24:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:10.135+0000 7f92bcc00700 1 -- 172.21.15.16:0/1887395875 wait complete. 2024-04-13T03:24:10.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:10.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:10 smithi016 ceph-mon[31772]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:10.826 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:10 smithi016 ceph-mon[31772]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:24:10.904 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2024-04-13T03:24:10.904 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:24:10.904 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2024-04-13T03:24:10.923 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:24:10.923 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:10.979 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2024-04-13T03:24:10.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch host add smithi188 2024-04-13T03:24:11.449 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:24:12.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:12 smithi016 ceph-mon[31772]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:12.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:12 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:12.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:12 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:12.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:12 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:12.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:12 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:12.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:12 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:24:12.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:12 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:12.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:12 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:13.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.434+0000 7f2113c00700 1 -- 172.21.15.16:0/3602102934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 msgr2=0x7f210c108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.434+0000 7f2113c00700 1 --2- 172.21.15.16:0/3602102934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 0x7f210c108920 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2100009910 tx=0x7f2100009c20 comp rx=0 tx=0).stop 2024-04-13T03:24:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.435+0000 7f2113c00700 1 -- 172.21.15.16:0/3602102934 shutdown_connections 2024-04-13T03:24:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.435+0000 7f2113c00700 1 --2- 172.21.15.16:0/3602102934 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 0x7f210c108920 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.435+0000 7f2113c00700 1 -- 172.21.15.16:0/3602102934 >> 172.21.15.16:0/3602102934 conn(0x7f210c076f10 msgr2=0x7f210c077320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.435+0000 7f2113c00700 1 -- 172.21.15.16:0/3602102934 shutdown_connections 2024-04-13T03:24:13.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.435+0000 7f2113c00700 1 -- 172.21.15.16:0/3602102934 wait complete. 2024-04-13T03:24:13.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.436+0000 7f2113c00700 1 Processor -- start 2024-04-13T03:24:13.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.436+0000 7f2113c00700 1 -- start start 2024-04-13T03:24:13.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.437+0000 7f2113c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 0x7f210c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:13.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.437+0000 7f2113c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f210c10ca80 con 0x7f210c108540 2024-04-13T03:24:13.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.437+0000 7f2111200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 0x7f210c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:13.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.437+0000 7f2111200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 0x7f210c19ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35888/0 (socket says 172.21.15.16:35888) 2024-04-13T03:24:13.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.437+0000 7f2111200700 1 -- 172.21.15.16:0/3630674745 learned_addr learned my addr 172.21.15.16:0/3630674745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:24:13.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.438+0000 7f2111200700 1 -- 172.21.15.16:0/3630674745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f210c19f240 con 0x7f210c108540 2024-04-13T03:24:13.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.438+0000 7f2111200700 1 --2- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 0x7f210c19ed00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2100006010 tx=0x7f2100005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:13.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.438+0000 7f2109600700 1 -- 172.21.15.16:0/3630674745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2100026070 con 0x7f210c108540 2024-04-13T03:24:13.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.438+0000 7f2109600700 1 -- 172.21.15.16:0/3630674745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f210000bc30 con 0x7f210c108540 2024-04-13T03:24:13.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.438+0000 7f2109600700 1 -- 172.21.15.16:0/3630674745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21000039d0 con 0x7f210c108540 2024-04-13T03:24:13.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.438+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f210c19f530 con 0x7f210c108540 2024-04-13T03:24:13.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.438+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f210c1a2e40 con 0x7f210c108540 2024-04-13T03:24:13.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.440+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f210c19f9f0 con 0x7f210c108540 2024-04-13T03:24:13.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.440+0000 7f2109600700 1 -- 172.21.15.16:0/3630674745 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2100029070 con 0x7f210c108540 2024-04-13T03:24:13.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.440+0000 7f2109600700 1 --2- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f20f40396b0 0x7f20f403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:13.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.441+0000 7f2109600700 1 -- 172.21.15.16:0/3630674745 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f210001e070 con 0x7f210c108540 2024-04-13T03:24:13.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.441+0000 7f210be00700 1 --2- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f20f40396b0 0x7f20f403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:13.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.442+0000 7f210be00700 1 --2- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f20f40396b0 0x7f20f403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f20f8006fd0 tx=0x7f20f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:13.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.448+0000 7f2109600700 1 -- 172.21.15.16:0/3630674745 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2100013070 con 0x7f210c108540 2024-04-13T03:24:13.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:13.686+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}) -- 0x7f210c103340 con 0x7f20f40396b0 2024-04-13T03:24:14.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:13 smithi016 ceph-mon[31772]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-13T03:24:14.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:13 smithi016 ceph-mon[31772]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:14.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:13 smithi016 ceph-mon[31772]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:15.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:14 smithi016 ceph-mon[31772]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:24:15.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:14 smithi016 ceph-mon[31772]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:14 smithi016 ceph-mon[31772]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:16.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:15 smithi016 ceph-mon[31772]: Deploying cephadm binary to smithi188 2024-04-13T03:24:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:16 smithi016 ceph-mon[31772]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.684+0000 7f2109600700 1 -- 172.21.15.16:0/3630674745 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f210c103340 con 0x7f20f40396b0 2024-04-13T03:24:18.687 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi188' with addr '172.21.15.188' 2024-04-13T03:24:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f20f40396b0 msgr2=0x7f20f403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 --2- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f20f40396b0 0x7f20f403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f20f8006fd0 tx=0x7f20f8006e40 comp rx=0 tx=0).stop 2024-04-13T03:24:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 msgr2=0x7f210c19ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 --2- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 0x7f210c19ed00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2100006010 tx=0x7f2100005030 comp rx=0 tx=0).stop 2024-04-13T03:24:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 shutdown_connections 2024-04-13T03:24:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 --2- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f20f40396b0 0x7f20f403bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 --2- 172.21.15.16:0/3630674745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f210c108540 0x7f210c19ed00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 >> 172.21.15.16:0/3630674745 conn(0x7f210c076f10 msgr2=0x7f210c10ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 shutdown_connections 2024-04-13T03:24:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:18.688+0000 7f2113c00700 1 -- 172.21.15.16:0/3630674745 wait complete. 2024-04-13T03:24:19.059 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:18 smithi016 ceph-mon[31772]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:19.059 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:19.060 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:19.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch host ls --format=json 2024-04-13T03:24:19.956 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:24:20.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:20 smithi016 ceph-mon[31772]: Added host smithi188 2024-04-13T03:24:20.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:20.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:21.466 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:21 smithi016 ceph-mon[31772]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.949+0000 7fe649800700 1 -- 172.21.15.16:0/3912807164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 msgr2=0x7fe644075260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.949+0000 7fe649800700 1 --2- 172.21.15.16:0/3912807164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 0x7fe644075260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe634009910 tx=0x7fe634009c20 comp rx=0 tx=0).stop 2024-04-13T03:24:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.949+0000 7fe649800700 1 -- 172.21.15.16:0/3912807164 shutdown_connections 2024-04-13T03:24:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.949+0000 7fe649800700 1 --2- 172.21.15.16:0/3912807164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 0x7fe644075260 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.949+0000 7fe649800700 1 -- 172.21.15.16:0/3912807164 >> 172.21.15.16:0/3912807164 conn(0x7fe6440fdff0 msgr2=0x7fe644100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.949+0000 7fe649800700 1 -- 172.21.15.16:0/3912807164 shutdown_connections 2024-04-13T03:24:21.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.949+0000 7fe649800700 1 -- 172.21.15.16:0/3912807164 wait complete. 2024-04-13T03:24:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.950+0000 7fe649800700 1 Processor -- start 2024-04-13T03:24:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.951+0000 7fe649800700 1 -- start start 2024-04-13T03:24:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.951+0000 7fe649800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 0x7fe64419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.951+0000 7fe649800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe64410ed60 con 0x7fe644074e80 2024-04-13T03:24:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.951+0000 7fe642a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 0x7fe64419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.951+0000 7fe642a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 0x7fe64419a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51198/0 (socket says 172.21.15.16:51198) 2024-04-13T03:24:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.951+0000 7fe642a00700 1 -- 172.21.15.16:0/2094472904 learned_addr learned my addr 172.21.15.16:0/2094472904 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:24:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.952+0000 7fe642a00700 1 -- 172.21.15.16:0/2094472904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64419a7d0 con 0x7fe644074e80 2024-04-13T03:24:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.952+0000 7fe642a00700 1 --2- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 0x7fe64419a290 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe634006010 tx=0x7fe634005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.952+0000 7fe63b400700 1 -- 172.21.15.16:0/2094472904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe634026070 con 0x7fe644074e80 2024-04-13T03:24:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.952+0000 7fe63b400700 1 -- 172.21.15.16:0/2094472904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe63400bc30 con 0x7fe644074e80 2024-04-13T03:24:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.952+0000 7fe63b400700 1 -- 172.21.15.16:0/2094472904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe634003ab0 con 0x7fe644074e80 2024-04-13T03:24:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.953+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64419aa60 con 0x7fe644074e80 2024-04-13T03:24:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.953+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64419aec0 con 0x7fe644074e80 2024-04-13T03:24:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.954+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe644076530 con 0x7fe644074e80 2024-04-13T03:24:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.954+0000 7fe63b400700 1 -- 172.21.15.16:0/2094472904 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe634029070 con 0x7fe644074e80 2024-04-13T03:24:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.955+0000 7fe63b400700 1 --2- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6280396b0 0x7fe62803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.955+0000 7fe63b400700 1 -- 172.21.15.16:0/2094472904 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe63401e070 con 0x7fe644074e80 2024-04-13T03:24:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.955+0000 7fe642000700 1 --2- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6280396b0 0x7fe62803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:21.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.956+0000 7fe642000700 1 --2- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6280396b0 0x7fe62803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe630006fd0 tx=0x7fe630006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:21.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:21.961+0000 7fe63b400700 1 -- 172.21.15.16:0/2094472904 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe634013070 con 0x7fe644074e80 2024-04-13T03:24:22.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.199+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe64419b5f0 con 0x7fe6280396b0 2024-04-13T03:24:22.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.202+0000 7fe63b400700 1 -- 172.21.15.16:0/2094472904 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fe64419b5f0 con 0x7fe6280396b0 2024-04-13T03:24:22.203 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:24:22.203 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2024-04-13T03:24:22.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.205+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6280396b0 msgr2=0x7fe62803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:22.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.205+0000 7fe649800700 1 --2- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6280396b0 0x7fe62803bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe630006fd0 tx=0x7fe630006e40 comp rx=0 tx=0).stop 2024-04-13T03:24:22.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.205+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 msgr2=0x7fe64419a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:22.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.205+0000 7fe649800700 1 --2- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 0x7fe64419a290 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe634006010 tx=0x7fe634005050 comp rx=0 tx=0).stop 2024-04-13T03:24:22.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.205+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 shutdown_connections 2024-04-13T03:24:22.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.205+0000 7fe649800700 1 --2- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6280396b0 0x7fe62803bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:22.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.205+0000 7fe649800700 1 --2- 172.21.15.16:0/2094472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe644074e80 0x7fe64419a290 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:22.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.205+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 >> 172.21.15.16:0/2094472904 conn(0x7fe6440fdff0 msgr2=0x7fe6441067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:22.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.205+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 shutdown_connections 2024-04-13T03:24:22.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:22.206+0000 7fe649800700 1 -- 172.21.15.16:0/2094472904 wait complete. 2024-04-13T03:24:22.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:22 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:22.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:22 smithi016 ceph-mon[31772]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:22.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:22 smithi016 ceph-mon[31772]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:24:22.890 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-13T03:24:22.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd crush tunables default 2024-04-13T03:24:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:24:25.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:24 smithi016 ceph-mon[31772]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:25.070 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:25.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:25.071 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.245+0000 7fe9aee00700 1 -- 172.21.15.16:0/90084901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 msgr2=0x7fe9a810aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:25.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.245+0000 7fe9aee00700 1 --2- 172.21.15.16:0/90084901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 0x7fe9a810aab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe998009910 tx=0x7fe998009c20 comp rx=0 tx=0).stop 2024-04-13T03:24:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.246+0000 7fe9aee00700 1 -- 172.21.15.16:0/90084901 shutdown_connections 2024-04-13T03:24:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.246+0000 7fe9aee00700 1 --2- 172.21.15.16:0/90084901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 0x7fe9a810aab0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.246+0000 7fe9aee00700 1 -- 172.21.15.16:0/90084901 >> 172.21.15.16:0/90084901 conn(0x7fe9a81001e0 msgr2=0x7fe9a8102600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.246+0000 7fe9aee00700 1 -- 172.21.15.16:0/90084901 shutdown_connections 2024-04-13T03:24:25.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.246+0000 7fe9aee00700 1 -- 172.21.15.16:0/90084901 wait complete. 2024-04-13T03:24:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.247+0000 7fe9aee00700 1 Processor -- start 2024-04-13T03:24:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.247+0000 7fe9aee00700 1 -- start start 2024-04-13T03:24:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.247+0000 7fe9aee00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 0x7fe9a8076fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.248+0000 7fe9aee00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a810ed10 con 0x7fe9a810a6d0 2024-04-13T03:24:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.248+0000 7fe9a7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 0x7fe9a8076fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.248+0000 7fe9a7e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 0x7fe9a8076fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51206/0 (socket says 172.21.15.16:51206) 2024-04-13T03:24:25.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.248+0000 7fe9a7e00700 1 -- 172.21.15.16:0/3913023334 learned_addr learned my addr 172.21.15.16:0/3913023334 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:24:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.248+0000 7fe9a7e00700 1 -- 172.21.15.16:0/3913023334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a807af20 con 0x7fe9a810a6d0 2024-04-13T03:24:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.249+0000 7fe9a7e00700 1 --2- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 0x7fe9a8076fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe998006010 tx=0x7fe998005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.249+0000 7fe9a4c00700 1 -- 172.21.15.16:0/3913023334 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe998026070 con 0x7fe9a810a6d0 2024-04-13T03:24:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.249+0000 7fe9a4c00700 1 -- 172.21.15.16:0/3913023334 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe99800bbd0 con 0x7fe9a810a6d0 2024-04-13T03:24:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.249+0000 7fe9a4c00700 1 -- 172.21.15.16:0/3913023334 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe998003910 con 0x7fe9a810a6d0 2024-04-13T03:24:25.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.249+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a8077500 con 0x7fe9a810a6d0 2024-04-13T03:24:25.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.249+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a80779c0 con 0x7fe9a810a6d0 2024-04-13T03:24:25.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.250+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a8105e10 con 0x7fe9a810a6d0 2024-04-13T03:24:25.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.251+0000 7fe9a4c00700 1 -- 172.21.15.16:0/3913023334 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe998029070 con 0x7fe9a810a6d0 2024-04-13T03:24:25.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.252+0000 7fe9a4c00700 1 --2- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe98c0396b0 0x7fe98c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:25.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.252+0000 7fe9a4c00700 1 -- 172.21.15.16:0/3913023334 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe99801e070 con 0x7fe9a810a6d0 2024-04-13T03:24:25.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.252+0000 7fe9a7400700 1 --2- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe98c0396b0 0x7fe98c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:25.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.253+0000 7fe9a7400700 1 --2- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe98c0396b0 0x7fe98c03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe994006fd0 tx=0x7fe994006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:25.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.258+0000 7fe9a4c00700 1 -- 172.21.15.16:0/3913023334 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe998013070 con 0x7fe9a810a6d0 2024-04-13T03:24:25.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.496+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fe9a807b0f0 con 0x7fe9a810a6d0 2024-04-13T03:24:25.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.801+0000 7fe9a4c00700 1 -- 172.21.15.16:0/3913023334 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fe99800fe20 con 0x7fe9a810a6d0 2024-04-13T03:24:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.806+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe98c0396b0 msgr2=0x7fe98c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.806+0000 7fe9aee00700 1 --2- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe98c0396b0 0x7fe98c03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe994006fd0 tx=0x7fe994006e40 comp rx=0 tx=0).stop 2024-04-13T03:24:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.806+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 msgr2=0x7fe9a8076fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.806+0000 7fe9aee00700 1 --2- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 0x7fe9a8076fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe998006010 tx=0x7fe998005030 comp rx=0 tx=0).stop 2024-04-13T03:24:25.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.806+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 shutdown_connections 2024-04-13T03:24:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.806+0000 7fe9aee00700 1 --2- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe98c0396b0 0x7fe98c03bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.806+0000 7fe9aee00700 1 --2- 172.21.15.16:0/3913023334 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a810a6d0 0x7fe9a8076fc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.806+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 >> 172.21.15.16:0/3913023334 conn(0x7fe9a81001e0 msgr2=0x7fe9a810c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.807+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 shutdown_connections 2024-04-13T03:24:25.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:25.807+0000 7fe9aee00700 1 -- 172.21.15.16:0/3913023334 wait complete. 2024-04-13T03:24:25.810 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-04-13T03:24:26.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:26.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:24:26.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:26.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:26.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:25 smithi016 ceph-mon[31772]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-13T03:24:26.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:25 smithi016 ceph-mon[31772]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:26.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:25 smithi016 ceph-mon[31772]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:26.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:25 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3913023334' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-13T03:24:26.572 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-04-13T03:24:26.572 INFO:tasks.cephadm:Adding mon.b on smithi119 2024-04-13T03:24:26.572 INFO:tasks.cephadm:Adding mon.c on smithi188 2024-04-13T03:24:26.573 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch apply mon '3;smithi016:172.21.15.16=a;smithi119:172.21.15.119=b;smithi188:172.21.15.188=c' 2024-04-13T03:24:26.797 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:27.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:26 smithi016 ceph-mon[31772]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:27.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:26 smithi016 ceph-mon[31772]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:24:27.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:26 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3913023334' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-13T03:24:27.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:26 smithi016 ceph-mon[31772]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:27.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:27.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:27.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:28.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.167+0000 7fd578f75700 1 -- 172.21.15.188:0/2419077599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 msgr2=0x7fd574108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:28.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.167+0000 7fd578f75700 1 --2- 172.21.15.188:0/2419077599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 0x7fd574108970 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd564009910 tx=0x7fd564009c20 comp rx=0 tx=0).stop 2024-04-13T03:24:28.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.167+0000 7fd578f75700 1 -- 172.21.15.188:0/2419077599 shutdown_connections 2024-04-13T03:24:28.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.167+0000 7fd578f75700 1 --2- 172.21.15.188:0/2419077599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 0x7fd574108970 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:28.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.167+0000 7fd578f75700 1 -- 172.21.15.188:0/2419077599 >> 172.21.15.188:0/2419077599 conn(0x7fd5740fdff0 msgr2=0x7fd574100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:28.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.167+0000 7fd578f75700 1 -- 172.21.15.188:0/2419077599 shutdown_connections 2024-04-13T03:24:28.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.167+0000 7fd578f75700 1 -- 172.21.15.188:0/2419077599 wait complete. 2024-04-13T03:24:28.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.168+0000 7fd578f75700 1 Processor -- start 2024-04-13T03:24:28.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.168+0000 7fd578f75700 1 -- start start 2024-04-13T03:24:28.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.168+0000 7fd578f75700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 0x7fd574197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:28.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.168+0000 7fd578f75700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd57410cb50 con 0x7fd574108590 2024-04-13T03:24:28.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.169+0000 7fd57259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 0x7fd574197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:28.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.169+0000 7fd57259c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 0x7fd574197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.188:41450/0 (socket says 172.21.15.188:41450) 2024-04-13T03:24:28.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.169+0000 7fd57259c700 1 -- 172.21.15.188:0/2809531102 learned_addr learned my addr 172.21.15.188:0/2809531102 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:24:28.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.169+0000 7fd57259c700 1 -- 172.21.15.188:0/2809531102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5741984d0 con 0x7fd574108590 2024-04-13T03:24:28.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.170+0000 7fd57259c700 1 --2- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 0x7fd574197f90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd564005850 tx=0x7fd564005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:28.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.170+0000 7fd5637fe700 1 -- 172.21.15.188:0/2809531102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd564026070 con 0x7fd574108590 2024-04-13T03:24:28.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.170+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd574198760 con 0x7fd574108590 2024-04-13T03:24:28.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.170+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd57419c860 con 0x7fd574108590 2024-04-13T03:24:28.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.170+0000 7fd5637fe700 1 -- 172.21.15.188:0/2809531102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd56400bc30 con 0x7fd574108590 2024-04-13T03:24:28.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.170+0000 7fd5637fe700 1 -- 172.21.15.188:0/2809531102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd564003680 con 0x7fd574108590 2024-04-13T03:24:28.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.171+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd554005330 con 0x7fd574108590 2024-04-13T03:24:28.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.172+0000 7fd5637fe700 1 -- 172.21.15.188:0/2809531102 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd564029070 con 0x7fd574108590 2024-04-13T03:24:28.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.172+0000 7fd5637fe700 1 --2- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd55c0396b0 0x7fd55c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:28.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.172+0000 7fd5637fe700 1 -- 172.21.15.188:0/2809531102 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd56401e070 con 0x7fd574108590 2024-04-13T03:24:28.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.172+0000 7fd571d9b700 1 --2- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd55c0396b0 0x7fd55c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:28.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.173+0000 7fd571d9b700 1 --2- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd55c0396b0 0x7fd55c03bb70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd568006fd0 tx=0x7fd568006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:28.188 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.187+0000 7fd5637fe700 1 -- 172.21.15.188:0/2809531102 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd564013070 con 0x7fd574108590 2024-04-13T03:24:28.398 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.397+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi119:172.21.15.119=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}) -- 0x7fd554000c80 con 0x7fd55c0396b0 2024-04-13T03:24:28.413 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.413+0000 7fd5637fe700 1 -- 172.21.15.188:0/2809531102 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd554000c80 con 0x7fd55c0396b0 2024-04-13T03:24:28.413 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2024-04-13T03:24:28.415 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.414+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd55c0396b0 msgr2=0x7fd55c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:28.416 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.414+0000 7fd578f75700 1 --2- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd55c0396b0 0x7fd55c03bb70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd568006fd0 tx=0x7fd568006e40 comp rx=0 tx=0).stop 2024-04-13T03:24:28.416 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.414+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 msgr2=0x7fd574197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:28.416 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.414+0000 7fd578f75700 1 --2- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 0x7fd574197f90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd564005850 tx=0x7fd564005030 comp rx=0 tx=0).stop 2024-04-13T03:24:28.416 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.415+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 shutdown_connections 2024-04-13T03:24:28.416 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.415+0000 7fd578f75700 1 --2- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd55c0396b0 0x7fd55c03bb70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:28.416 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.415+0000 7fd578f75700 1 --2- 172.21.15.188:0/2809531102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd574108590 0x7fd574197f90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:28.416 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.415+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 >> 172.21.15.188:0/2809531102 conn(0x7fd5740fdff0 msgr2=0x7fd5740ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:28.416 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.415+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 shutdown_connections 2024-04-13T03:24:28.416 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:28.415+0000 7fd578f75700 1 -- 172.21.15.188:0/2809531102 wait complete. 2024-04-13T03:24:29.039 DEBUG:teuthology.orchestra.run.smithi119:mon.b> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.b.service 2024-04-13T03:24:29.042 DEBUG:teuthology.orchestra.run.smithi188:mon.c> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.c.service 2024-04-13T03:24:29.044 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-13T03:24:29.044 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi119:172.21.15.119=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi119:172.21.15.119=b;smithi188:172.21.15.188=c;count:3 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:30.104 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:30.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:30 smithi016 ceph-mon[31772]: Deploying daemon mon.c on smithi188 2024-04-13T03:24:31.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:31 smithi016 ceph-mon[31772]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:32.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.770+0000 7fbb01cfd700 1 -- 172.21.15.188:0/568542461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 msgr2=0x7fbafc101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:32.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.770+0000 7fbb01cfd700 1 --2- 172.21.15.188:0/568542461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 0x7fbafc101a40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400b0a0 tx=0x7fbaf400b3b0 comp rx=0 tx=0).stop 2024-04-13T03:24:32.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.770+0000 7fbb01cfd700 1 -- 172.21.15.188:0/568542461 shutdown_connections 2024-04-13T03:24:32.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.770+0000 7fbb01cfd700 1 --2- 172.21.15.188:0/568542461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 0x7fbafc101a40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:32.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.770+0000 7fbb01cfd700 1 -- 172.21.15.188:0/568542461 >> 172.21.15.188:0/568542461 conn(0x7fbafc076f10 msgr2=0x7fbafc077320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:32.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.770+0000 7fbb01cfd700 1 -- 172.21.15.188:0/568542461 shutdown_connections 2024-04-13T03:24:32.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.770+0000 7fbb01cfd700 1 -- 172.21.15.188:0/568542461 wait complete. 2024-04-13T03:24:32.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.771+0000 7fbb01cfd700 1 Processor -- start 2024-04-13T03:24:32.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.771+0000 7fbb01cfd700 1 -- start start 2024-04-13T03:24:32.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.771+0000 7fbb01cfd700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 0x7fbafc198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:32.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.771+0000 7fbb01cfd700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbafc10b9e0 con 0x7fbafc101660 2024-04-13T03:24:32.774 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.772+0000 7fbafb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 0x7fbafc198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:32.774 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.772+0000 7fbafb7fe700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 0x7fbafc198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.188:48730/0 (socket says 172.21.15.188:48730) 2024-04-13T03:24:32.774 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.772+0000 7fbafb7fe700 1 -- 172.21.15.188:0/1818892036 learned_addr learned my addr 172.21.15.188:0/1818892036 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:24:32.775 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.772+0000 7fbafb7fe700 1 -- 172.21.15.188:0/1818892036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbafc199370 con 0x7fbafc101660 2024-04-13T03:24:32.775 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.773+0000 7fbafb7fe700 1 --2- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 0x7fbafc198e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4015040 tx=0x7fbaf4003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:32.775 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.773+0000 7fbaf8ff9700 1 -- 172.21.15.188:0/1818892036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaf4017040 con 0x7fbafc101660 2024-04-13T03:24:32.775 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.773+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbafc199600 con 0x7fbafc101660 2024-04-13T03:24:32.775 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.773+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbafc199ac0 con 0x7fbafc101660 2024-04-13T03:24:32.775 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.773+0000 7fbaf8ff9700 1 -- 172.21.15.188:0/1818892036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbaf4022070 con 0x7fbafc101660 2024-04-13T03:24:32.775 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.773+0000 7fbaf8ff9700 1 -- 172.21.15.188:0/1818892036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaf4027ba0 con 0x7fbafc101660 2024-04-13T03:24:32.775 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.775+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc005330 con 0x7fbafc101660 2024-04-13T03:24:32.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.775+0000 7fbaf8ff9700 1 -- 172.21.15.188:0/1818892036 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbaf4027460 con 0x7fbafc101660 2024-04-13T03:24:32.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.775+0000 7fbaf8ff9700 1 --2- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fbae8039650 0x7fbae803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:32.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.775+0000 7fbaf8ff9700 1 -- 172.21.15.188:0/1818892036 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbaf400f050 con 0x7fbafc101660 2024-04-13T03:24:32.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.776+0000 7fbafaffd700 1 --2- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fbae8039650 0x7fbae803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:32.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.777+0000 7fbafaffd700 1 --2- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fbae8039650 0x7fbae803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006fd0 tx=0x7fbae4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:32.792 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:32.791+0000 7fbaf8ff9700 1 -- 172.21.15.188:0/1818892036 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbaf4007e20 con 0x7fbafc101660 2024-04-13T03:24:33.087 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.085+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fbadc005620 con 0x7fbafc101660 2024-04-13T03:24:33.087 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.086+0000 7fbaf8ff9700 1 -- 172.21.15.188:0/1818892036 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fbaf40102f0 con 0x7fbafc101660 2024-04-13T03:24:33.087 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:24:33.087 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","modified":"2024-04-13T03:22:18.286080Z","created":"2024-04-13T03:22:18.286080Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fbae8039650 msgr2=0x7fbae803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 --2- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fbae8039650 0x7fbae803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006fd0 tx=0x7fbae4006e40 comp rx=0 tx=0).stop 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 msgr2=0x7fbafc198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 --2- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 0x7fbafc198e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4015040 tx=0x7fbaf4003d50 comp rx=0 tx=0).stop 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 shutdown_connections 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 --2- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fbae8039650 0x7fbae803bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 --2- 172.21.15.188:0/1818892036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbafc101660 0x7fbafc198e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 >> 172.21.15.188:0/1818892036 conn(0x7fbafc076f10 msgr2=0x7fbafc102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 shutdown_connections 2024-04-13T03:24:33.089 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:33.088+0000 7fbb01cfd700 1 -- 172.21.15.188:0/1818892036 wait complete. 2024-04-13T03:24:33.090 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-13T03:24:33.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 podman[36367]: 2024-04-13 03:24:33.148784606 +0000 UTC m=+0.361481828 container start 3b30980af288df4891c0c16fb7c21d0a963f3b094badaadf0f652cbfc3a4323b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: set uid:gid to 167:167 (ceph:ceph) 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev), process ceph-mon, pid 2 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pidfile_write: ignore empty --pid-file 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: load: jerasure load: lrc 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: RocksDB version: 7.9.2 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Git sha 0 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Compile date 2024-04-12 04:47:35 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: DB SUMMARY 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: DB Session ID: BS5K3W46WPYA46WQP9JB 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: CURRENT file: CURRENT 2024-04-13T03:24:33.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: IDENTITY file: IDENTITY 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.error_if_exists: 0 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.create_if_missing: 0 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.paranoid_checks: 1 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.env: 0x55e4aae7acc0 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.fs: PosixFileSystem 2024-04-13T03:24:33.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.info_log: 0x55e4acc73be0 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_file_opening_threads: 16 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.statistics: (nil) 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.use_fsync: 0 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_log_file_size: 0 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.keep_log_file_num: 1000 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.recycle_log_file_num: 0 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.allow_fallocate: 1 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.allow_mmap_reads: 0 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.allow_mmap_writes: 0 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.use_direct_reads: 0 2024-04-13T03:24:33.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.create_missing_column_families: 0 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.db_log_dir: 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.wal_dir: 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.advise_random_on_open: 1 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.db_write_buffer_size: 0 2024-04-13T03:24:33.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.write_buffer_manager: 0x55e4acc441e0 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.rate_limiter: (nil) 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.wal_recovery_mode: 2 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.enable_thread_tracking: 0 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.enable_pipelined_write: 0 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.unordered_write: 0 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-13T03:24:33.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.row_cache: None 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.wal_filter: None 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.allow_ingest_behind: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.two_write_queues: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.manual_wal_flush: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.wal_compression: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.atomic_flush: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.log_readahead_size: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.best_efforts_recovery: 0 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-13T03:24:33.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-13T03:24:33.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.allow_data_in_errors: 0 2024-04-13T03:24:33.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.db_host_id: __hostname__ 2024-04-13T03:24:33.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-13T03:24:33.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_background_jobs: 2 2024-04-13T03:24:33.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_background_compactions: -1 2024-04-13T03:24:33.523 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_subcompactions: 1 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_total_wal_size: 0 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_open_files: -1 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bytes_per_sync: 0 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_readahead_size: 0 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_background_flushes: -1 2024-04-13T03:24:33.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Compression algorithms supported: 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: kZSTD supported: 0 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: kXpressCompression supported: 0 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: kLZ4HCCompression supported: 1 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: kZlibCompression supported: 1 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: kSnappyCompression supported: 1 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: kLZ4Compression supported: 1 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: kBZip2Compression supported: 0 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-13T03:24:33.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.merge_operator: 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_filter: None 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_filter_factory: None 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.sst_partitioner_factory: None 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e4acc73dc0) 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks: 1 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: pin_top_level_index_and_filter: 1 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_type: 0 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_index_type: 0 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_shortening: 1 2024-04-13T03:24:33.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: checksum: 4 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: no_block_cache: 0 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache: 0x55e4acc77090 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_name: BinnedLRUCache 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_options: 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: capacity : 536870912 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_shard_bits : 4 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: strict_capacity_limit : 0 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: high_pri_pool_ratio: 0.000 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_cache_compressed: (nil) 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: persistent_cache: (nil) 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size: 4096 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_size_deviation: 10 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_restart_interval: 16 2024-04-13T03:24:33.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: index_block_restart_interval: 1 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: metadata_block_size: 4096 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: partition_filters: 0 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: use_delta_encoding: 1 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: filter_policy: bloomfilter 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: whole_key_filtering: 1 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: verify_compression: 0 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: read_amp_bytes_per_bit: 0 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: format_version: 5 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_index_compression: 1 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: block_align: 0 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: max_auto_readahead_size: 262144 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: prepopulate_block_cache: 0 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: initial_auto_readahead_size: 8192 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: num_file_reads_for_auto_readahead: 2 2024-04-13T03:24:33.528 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.write_buffer_size: 33554432 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_write_buffer_number: 2 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression: NoCompression 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression: Disabled 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.prefix_extractor: nullptr 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.num_levels: 7 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-13T03:24:33.529 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression_opts.level: 32767 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression_opts.strategy: 0 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression_opts.enabled: false 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-13T03:24:33.530 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.target_file_size_base: 67108864 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-13T03:24:33.531 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.arena_block_size: 1048576 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.disable_auto_compactions: 0 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-13T03:24:33.532 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.inplace_update_support: 0 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.bloom_locality: 0 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.max_successive_merges: 0 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.paranoid_file_checks: 0 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.force_consistency_checks: 1 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.report_bg_io_stats: 0 2024-04-13T03:24:33.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.ttl: 2592000 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.enable_blob_files: false 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.min_blob_size: 0 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.blob_file_size: 268435456 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.blob_file_starting_level: 0 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-13T03:24:33.534 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7fec3a0c-9bb9-42a5-b68d-c648093606b1 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712978673178630, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712978673179043, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712978673, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7fec3a0c-9bb9-42a5-b68d-c648093606b1", "db_session_id": "BS5K3W46WPYA46WQP9JB", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712978673179125, "job": 1, "event": "recovery_finished"} 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e4acd66000 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: DB pointer 0x55e4acd50000 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** DB Stats ** 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-13T03:24:33.535 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-13T03:24:33.536 INFO:journalctl@ceph.mon.c.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-13T03:24:33.537 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache BinnedLRUCache@0x55e4acc77090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: using public_addr v2:172.21.15.188:0/0 -> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: starting mon.c rank -1 at public addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] at bind addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(???) e0 preinit fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).mds e1 new map 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout: e1 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout: legacy client fscid: -1 2024-04-13T03:24:33.538 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout: No filesystems configured 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Added host smithi119 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.539 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:33.540 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Deploying cephadm binary to smithi188 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Added host smithi188 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.541 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3913023334' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.542 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3913023334' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi119:172.21.15.119=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi119:172.21.15.119=b;smithi188:172.21.15.188=c;count:3 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:33.543 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: Deploying daemon mon.c on smithi188 2024-04-13T03:24:33.544 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:33.544 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 ceph-mon[36398]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-13T03:24:33.544 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 bash[36367]: 3b30980af288df4891c0c16fb7c21d0a963f3b094badaadf0f652cbfc3a4323b 2024-04-13T03:24:33.544 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:33 smithi188 systemd[1]: Started Ceph mon.c for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:24:34.768 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-13T03:24:34.768 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-13T03:24:35.016 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:24:38.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:24:38.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: mon.a calling monitor election 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: mon.c calling monitor election 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: monmap epoch 2 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: last_changed 2024-04-13T03:24:33.213674+0000 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:24:38.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: min_mon_release 19 (squid) 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: election_strategy: 1 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: fsmap 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: mgrmap e13: x(active, since 58s) 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: overall HEALTH_OK 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:38.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: mon.a calling monitor election 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: mon.c calling monitor election 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: monmap epoch 2 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: last_changed 2024-04-13T03:24:33.213674+0000 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: min_mon_release 19 (squid) 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: election_strategy: 1 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: fsmap 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: mgrmap e13: x(active, since 58s) 2024-04-13T03:24:38.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: overall HEALTH_OK 2024-04-13T03:24:38.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:38.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:38.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:38.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:38.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:39.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:39 smithi188 ceph-mon[36398]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:39.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:39 smithi188 ceph-mon[36398]: Deploying daemon mon.b on smithi119 2024-04-13T03:24:39.358 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:39 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:39.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.357+0000 7f44239e9700 1 -- 172.21.15.188:0/3821957080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44080035c0 msgr2=0x7f4408005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:39.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.357+0000 7f44239e9700 1 --2- 172.21.15.188:0/3821957080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44080035c0 0x7f4408005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f441c06ad00 tx=0x7f441800feb0 comp rx=0 tx=0).stop 2024-04-13T03:24:39.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.357+0000 7f44239e9700 1 -- 172.21.15.188:0/3821957080 shutdown_connections 2024-04-13T03:24:39.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.357+0000 7f44239e9700 1 --2- 172.21.15.188:0/3821957080 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44080035c0 0x7f4408005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:39.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.357+0000 7f44239e9700 1 -- 172.21.15.188:0/3821957080 >> 172.21.15.188:0/3821957080 conn(0x7f441c0fcfb0 msgr2=0x7f441c0ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:39.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.357+0000 7f44239e9700 1 -- 172.21.15.188:0/3821957080 shutdown_connections 2024-04-13T03:24:39.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.357+0000 7f44239e9700 1 -- 172.21.15.188:0/3821957080 wait complete. 2024-04-13T03:24:39.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f44239e9700 1 Processor -- start 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f44239e9700 1 -- start start 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f44239e9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 0x7f441c101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f44239e9700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f441c1019c0 0x7f441c101e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f44239e9700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f441c10a660 con 0x7f441c1019c0 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f44239e9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f441c10a4e0 con 0x7f44080035c0 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f4421785700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 0x7f441c101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f4421785700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 0x7f441c101480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:34334/0 (socket says 172.21.15.188:34334) 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f4421785700 1 -- 172.21.15.188:0/1775039321 learned_addr learned my addr 172.21.15.188:0/1775039321 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f4421785700 1 -- 172.21.15.188:0/1775039321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 msgr2=0x7f441c101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f4421785700 1 -- 172.21.15.188:0/1775039321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 msgr2=0x7f441c101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f4421785700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 0x7f441c101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f4421785700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 0x7f441c101480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.358+0000 7f4420f84700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f441c1019c0 0x7f441c101e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.359+0000 7f4420f84700 1 -- 172.21.15.188:0/1775039321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 msgr2=0x7f441c101480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:24:39.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.359+0000 7f4420f84700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 0x7f441c101480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:39.362 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.359+0000 7f4420f84700 1 -- 172.21.15.188:0/1775039321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441c1033a0 con 0x7f441c1019c0 2024-04-13T03:24:39.362 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.359+0000 7f4420f84700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f441c1019c0 0x7f441c101e40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f440c00b4d0 tx=0x7f440c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.360+0000 7f44127fc700 1 -- 172.21.15.188:0/1775039321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f440c011630 con 0x7f441c1019c0 2024-04-13T03:24:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.360+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f441c1a4480 con 0x7f441c1019c0 2024-04-13T03:24:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.360+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f441c1a49c0 con 0x7f441c1019c0 2024-04-13T03:24:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.360+0000 7f44127fc700 1 -- 172.21.15.188:0/1775039321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f440c011cb0 con 0x7f441c1019c0 2024-04-13T03:24:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.360+0000 7f44127fc700 1 -- 172.21.15.188:0/1775039321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f440c0193b0 con 0x7f441c1019c0 2024-04-13T03:24:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.361+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4408002b60 con 0x7f441c1019c0 2024-04-13T03:24:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.362+0000 7f44127fc700 1 -- 172.21.15.188:0/1775039321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f440c018020 con 0x7f441c1019c0 2024-04-13T03:24:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.362+0000 7f44127fc700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f44000393f0 0x7f440003b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:39.363 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.362+0000 7f44127fc700 1 -- 172.21.15.188:0/1775039321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f440c04c440 con 0x7f441c1019c0 2024-04-13T03:24:39.364 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.362+0000 7f4421785700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f44000393f0 0x7f440003b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:39.364 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.363+0000 7f4421785700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f44000393f0 0x7f440003b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f441800fa60 tx=0x7f4418005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:39.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.377+0000 7f44127fc700 1 -- 172.21.15.188:0/1775039321 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f440c01f020 con 0x7f441c1019c0 2024-04-13T03:24:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:39 smithi016 ceph-mon[31772]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:39 smithi016 ceph-mon[31772]: Deploying daemon mon.b on smithi119 2024-04-13T03:24:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:39.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.655+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4408002e50 con 0x7f441c1019c0 2024-04-13T03:24:39.656 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.656+0000 7f44127fc700 1 -- 172.21.15.188:0/1775039321 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f440c021530 con 0x7f441c1019c0 2024-04-13T03:24:39.657 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:24:39.657 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","modified":"2024-04-13T03:24:33.213674Z","created":"2024-04-13T03:22:18.286080Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-13T03:24:39.659 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f44000393f0 msgr2=0x7f440003b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:39.659 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f44000393f0 0x7f440003b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f441800fa60 tx=0x7f4418005960 comp rx=0 tx=0).stop 2024-04-13T03:24:39.659 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f441c1019c0 msgr2=0x7f441c101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:39.659 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f441c1019c0 0x7f441c101e40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f440c00b4d0 tx=0x7f440c00b890 comp rx=0 tx=0).stop 2024-04-13T03:24:39.660 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 shutdown_connections 2024-04-13T03:24:39.660 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f44000393f0 0x7f440003b8b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:39.660 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f44080035c0 0x7f441c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:39.660 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 --2- 172.21.15.188:0/1775039321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f441c1019c0 0x7f441c101e40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:39.660 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 >> 172.21.15.188:0/1775039321 conn(0x7f441c0fcfb0 msgr2=0x7f441c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:39.660 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 shutdown_connections 2024-04-13T03:24:39.660 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:39.658+0000 7f44239e9700 1 -- 172.21.15.188:0/1775039321 wait complete. 2024-04-13T03:24:39.660 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2024-04-13T03:24:40.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:40 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/1775039321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-13T03:24:40.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:40 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/1775039321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-13T03:24:41.254 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-13T03:24:41.254 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-13T03:24:41.499 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:24:41.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:41 smithi016 ceph-mon[31772]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:41.618 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:41 smithi188 ceph-mon[36398]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:41.969 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 podman[35814]: 2024-04-13 03:24:41.945656448 +0000 UTC m=+0.356161026 container start a9319192e0cf8b6fa6cec5c530f78935af16ab4e192371561c81ece95bb7c572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-13T03:24:42.271 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: set uid:gid to 167:167 (ceph:ceph) 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev), process ceph-mon, pid 2 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pidfile_write: ignore empty --pid-file 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: load: jerasure load: lrc 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: RocksDB version: 7.9.2 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Git sha 0 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Compile date 2024-04-12 04:47:35 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: DB SUMMARY 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: DB Session ID: E3EIXBHSHY6NTMEV3B03 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: CURRENT file: CURRENT 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: IDENTITY file: IDENTITY 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.error_if_exists: 0 2024-04-13T03:24:42.272 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.create_if_missing: 0 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.paranoid_checks: 1 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.env: 0x557496368cc0 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.fs: PosixFileSystem 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.info_log: 0x55749799bbe0 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_file_opening_threads: 16 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.statistics: (nil) 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.use_fsync: 0 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_log_file_size: 0 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.keep_log_file_num: 1000 2024-04-13T03:24:42.273 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.recycle_log_file_num: 0 2024-04-13T03:24:42.274 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.allow_fallocate: 1 2024-04-13T03:24:42.274 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.allow_mmap_reads: 0 2024-04-13T03:24:42.274 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.allow_mmap_writes: 0 2024-04-13T03:24:42.274 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.use_direct_reads: 0 2024-04-13T03:24:42.274 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-13T03:24:42.274 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.create_missing_column_families: 0 2024-04-13T03:24:42.274 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.db_log_dir: 2024-04-13T03:24:42.274 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.wal_dir: 2024-04-13T03:24:42.274 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-13T03:24:42.276 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-13T03:24:42.276 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-13T03:24:42.276 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-13T03:24:42.276 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-13T03:24:42.276 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-13T03:24:42.276 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.advise_random_on_open: 1 2024-04-13T03:24:42.276 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.db_write_buffer_size: 0 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.write_buffer_manager: 0x55749796c1e0 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.rate_limiter: (nil) 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.wal_recovery_mode: 2 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.enable_thread_tracking: 0 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.enable_pipelined_write: 0 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.unordered_write: 0 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.row_cache: None 2024-04-13T03:24:42.277 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.wal_filter: None 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.allow_ingest_behind: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.two_write_queues: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.manual_wal_flush: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.wal_compression: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.atomic_flush: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.log_readahead_size: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.best_efforts_recovery: 0 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-13T03:24:42.278 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.allow_data_in_errors: 0 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.db_host_id: __hostname__ 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_background_jobs: 2 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_background_compactions: -1 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_subcompactions: 1 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_total_wal_size: 0 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_open_files: -1 2024-04-13T03:24:42.279 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bytes_per_sync: 0 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_readahead_size: 0 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_background_flushes: -1 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Compression algorithms supported: 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: kZSTD supported: 0 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: kXpressCompression supported: 0 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: kLZ4HCCompression supported: 1 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: kZlibCompression supported: 1 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: kSnappyCompression supported: 1 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: kLZ4Compression supported: 1 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: kBZip2Compression supported: 0 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-13T03:24:42.280 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.merge_operator: 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_filter: None 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_filter_factory: None 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.sst_partitioner_factory: None 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55749799bdc0) 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-04-13T03:24:42.281 INFO:journalctl@ceph.mon.b.smithi119.stdout: index_type: 0 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: data_block_index_type: 0 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: index_shortening: 1 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: checksum: 4 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: no_block_cache: 0 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache: 0x55749799f090 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache_options: 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: capacity : 536870912 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: num_shard_bits : 4 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: strict_capacity_limit : 0 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache_compressed: (nil) 2024-04-13T03:24:42.282 INFO:journalctl@ceph.mon.b.smithi119.stdout: persistent_cache: (nil) 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_size: 4096 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_size_deviation: 10 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_restart_interval: 16 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: index_block_restart_interval: 1 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: metadata_block_size: 4096 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: partition_filters: 0 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: use_delta_encoding: 1 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: filter_policy: bloomfilter 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: whole_key_filtering: 1 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: verify_compression: 0 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: format_version: 5 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: enable_index_compression: 1 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_align: 0 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: max_auto_readahead_size: 262144 2024-04-13T03:24:42.283 INFO:journalctl@ceph.mon.b.smithi119.stdout: prepopulate_block_cache: 0 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout: initial_auto_readahead_size: 8192 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout: num_file_reads_for_auto_readahead: 2 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.write_buffer_size: 33554432 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_write_buffer_number: 2 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression: NoCompression 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression: Disabled 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.prefix_extractor: nullptr 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.num_levels: 7 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-13T03:24:42.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression_opts.level: 32767 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression_opts.strategy: 0 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression_opts.enabled: false 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-13T03:24:42.285 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.target_file_size_base: 67108864 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-13T03:24:42.286 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.arena_block_size: 1048576 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.disable_auto_compactions: 0 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-13T03:24:42.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.inplace_update_support: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.bloom_locality: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.max_successive_merges: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.paranoid_file_checks: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.force_consistency_checks: 1 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.report_bg_io_stats: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.ttl: 2592000 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-13T03:24:42.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.enable_blob_files: false 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.min_blob_size: 0 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.blob_file_size: 268435456 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.blob_file_starting_level: 0 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-13T03:24:42.289 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6e107dbc-d513-4908-990f-e5b36872b09f 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712978681975270, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712978681975692, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712978681, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6e107dbc-d513-4908-990f-e5b36872b09f", "db_session_id": "E3EIXBHSHY6NTMEV3B03", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712978681975775, "job": 1, "event": "recovery_finished"} 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557497a8e000 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: DB pointer 0x557497a78000 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** DB Stats ** 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-13T03:24:42.290 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** Compaction Stats [default] ** 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** Compaction Stats [default] ** 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-13T03:24:42.291 INFO:journalctl@ceph.mon.b.smithi119.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-13T03:24:42.292 INFO:journalctl@ceph.mon.b.smithi119.stdout: Block cache BinnedLRUCache@0x55749799f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: using public_addr v2:172.21.15.119:0/0 -> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: starting mon.b rank -1 at public addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] at bind addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(???) e0 preinit fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).mds e1 new map 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout: e1 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout: legacy client fscid: -1 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-13T03:24:42.293 INFO:journalctl@ceph.mon.b.smithi119.stdout: No filesystems configured 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Added host smithi119 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:42.295 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Deploying cephadm binary to smithi188 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Added host smithi188 2024-04-13T03:24:42.296 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:42.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3913023334' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3913023334' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi119:172.21.15.119=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi119:172.21.15.119=b;smithi188:172.21.15.188=c;count:3 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Deploying daemon mon.c on smithi188 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.a calling monitor election 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.c calling monitor election 2024-04-13T03:24:42.299 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: monmap epoch 2 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: last_changed 2024-04-13T03:24:33.213674+0000 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: min_mon_release 19 (squid) 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: election_strategy: 1 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: fsmap 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mgrmap e13: x(active, since 58s) 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: overall HEALTH_OK 2024-04-13T03:24:42.300 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: Deploying daemon mon.b on smithi119 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/1775039321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:41 smithi119 ceph-mon[35835]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:42 smithi119 bash[35814]: a9319192e0cf8b6fa6cec5c530f78935af16ab4e192371561c81ece95bb7c572 2024-04-13T03:24:42.301 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:42 smithi119 systemd[1]: Started Ceph mon.b for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:24:47.339 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.339+0000 7fc509c5c700 1 -- 172.21.15.188:0/3348457190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc504108680 msgr2=0x7fc4f0005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:47.340 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.339+0000 7fc509c5c700 1 --2- 172.21.15.188:0/3348457190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc504108680 0x7fc4f0005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00fc30 tx=0x7fc4fc00ff40 comp rx=0 tx=0).stop 2024-04-13T03:24:47.340 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.339+0000 7fc509c5c700 1 -- 172.21.15.188:0/3348457190 shutdown_connections 2024-04-13T03:24:47.340 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.339+0000 7fc509c5c700 1 --2- 172.21.15.188:0/3348457190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc504108680 0x7fc4f0005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:47.340 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.339+0000 7fc509c5c700 1 -- 172.21.15.188:0/3348457190 >> 172.21.15.188:0/3348457190 conn(0x7fc5040fe0a0 msgr2=0x7fc5041004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:47.340 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.340+0000 7fc509c5c700 1 -- 172.21.15.188:0/3348457190 shutdown_connections 2024-04-13T03:24:47.340 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.340+0000 7fc509c5c700 1 -- 172.21.15.188:0/3348457190 wait complete. 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: mon.c calling monitor election 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: mon.a calling monitor election 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:47.341 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: monmap epoch 3 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: last_changed 2024-04-13T03:24:41.998298+0000 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: min_mon_release 19 (squid) 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: election_strategy: 1 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: fsmap 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:47.342 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: mgrmap e13: x(active, since 67s) 2024-04-13T03:24:47.343 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: overall HEALTH_OK 2024-04-13T03:24:47.343 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:47.343 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:47.345 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.340+0000 7fc509c5c700 1 Processor -- start 2024-04-13T03:24:47.345 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.341+0000 7fc509c5c700 1 -- start start 2024-04-13T03:24:47.345 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.341+0000 7fc509c5c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc504108680 0x7fc50419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:47.345 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.341+0000 7fc509c5c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50419f4c0 0x7fc50419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:47.345 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.341+0000 7fc509c5c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 0x7fc50419acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:47.345 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.341+0000 7fc509c5c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc50410a660 con 0x7fc504108680 2024-04-13T03:24:47.345 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.341+0000 7fc509c5c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc50410a4e0 con 0x7fc50419f4c0 2024-04-13T03:24:47.345 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.341+0000 7fc509c5c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc50410a7e0 con 0x7fc50419a820 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.341+0000 7fc503fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 0x7fc50419acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc503fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 0x7fc50419acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:56036/0 (socket says 172.21.15.188:56036) 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc503fff700 1 -- 172.21.15.188:0/848288149 learned_addr learned my addr 172.21.15.188:0/848288149 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc5037fe700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc504108680 0x7fc50419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc503fff700 1 -- 172.21.15.188:0/848288149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 msgr2=0x7fc50419acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc502ffd700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50419f4c0 0x7fc50419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc503fff700 1 -- 172.21.15.188:0/848288149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 msgr2=0x7fc50419acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc503fff700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 0x7fc50419acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc503fff700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 0x7fc50419acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc502ffd700 1 -- 172.21.15.188:0/848288149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50419f4c0 msgr2=0x7fc50419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc502ffd700 1 -- 172.21.15.188:0/848288149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50419f4c0 msgr2=0x7fc50419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-13T03:24:47.346 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc502ffd700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50419f4c0 0x7fc50419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.342+0000 7fc502ffd700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50419f4c0 0x7fc50419a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.343+0000 7fc5037fe700 1 -- 172.21.15.188:0/848288149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 msgr2=0x7fc50419acd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.343+0000 7fc5037fe700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 0x7fc50419acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.343+0000 7fc5037fe700 1 -- 172.21.15.188:0/848288149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50419f4c0 msgr2=0x7fc50419a2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.343+0000 7fc5037fe700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50419f4c0 0x7fc50419a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.343+0000 7fc5037fe700 1 -- 172.21.15.188:0/848288149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5041a6f00 con 0x7fc504108680 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.343+0000 7fc5037fe700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc504108680 0x7fc50419ef80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc01f040 tx=0x7fc4fc012d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.344+0000 7fc500ff9700 1 -- 172.21.15.188:0/848288149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc02b070 con 0x7fc504108680 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.344+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5041a7130 con 0x7fc504108680 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.344+0000 7fc500ff9700 1 -- 172.21.15.188:0/848288149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4fc01db70 con 0x7fc504108680 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.344+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5041a7710 con 0x7fc504108680 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.344+0000 7fc500ff9700 1 -- 172.21.15.188:0/848288149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc035060 con 0x7fc504108680 2024-04-13T03:24:47.347 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.345+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc504064110 con 0x7fc504108680 2024-04-13T03:24:47.348 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.346+0000 7fc500ff9700 1 -- 172.21.15.188:0/848288149 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc4fc02e070 con 0x7fc504108680 2024-04-13T03:24:47.348 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.346+0000 7fc500ff9700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc4e4039ae0 0x7fc4e403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:47.348 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.346+0000 7fc500ff9700 1 -- 172.21.15.188:0/848288149 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc4fc04f660 con 0x7fc504108680 2024-04-13T03:24:47.349 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.346+0000 7fc502ffd700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc4e4039ae0 0x7fc4e403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:47.349 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.348+0000 7fc502ffd700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc4e4039ae0 0x7fc4e403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009850 tx=0x7fc4ec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: mon.c calling monitor election 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: mon.a calling monitor election 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: monmap epoch 3 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: last_changed 2024-04-13T03:24:41.998298+0000 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: min_mon_release 19 (squid) 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: election_strategy: 1 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: fsmap 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: mgrmap e13: x(active, since 67s) 2024-04-13T03:24:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: overall HEALTH_OK 2024-04-13T03:24:47.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:47.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:47.364 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.365+0000 7fc500ff9700 1 -- 172.21.15.188:0/848288149 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4fc010e40 con 0x7fc504108680 2024-04-13T03:24:47.662 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.662+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc50419b760 con 0x7fc504108680 2024-04-13T03:24:47.663 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.663+0000 7fc500ff9700 1 -- 172.21.15.188:0/848288149 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7fc4fc004060 con 0x7fc504108680 2024-04-13T03:24:47.663 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-13T03:24:47.663 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":3,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","modified":"2024-04-13T03:24:41.998298Z","created":"2024-04-13T03:22:18.286080Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-13T03:24:47.665 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc4e4039ae0 msgr2=0x7fc4e403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:47.665 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc4e4039ae0 0x7fc4e403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009850 tx=0x7fc4ec008040 comp rx=0 tx=0).stop 2024-04-13T03:24:47.665 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc504108680 msgr2=0x7fc50419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:47.665 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc504108680 0x7fc50419ef80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc01f040 tx=0x7fc4fc012d80 comp rx=0 tx=0).stop 2024-04-13T03:24:47.665 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 shutdown_connections 2024-04-13T03:24:47.665 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc4e4039ae0 0x7fc4e403bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:47.666 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc50419a820 0x7fc50419acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:47.666 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc504108680 0x7fc50419ef80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:47.666 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 --2- 172.21.15.188:0/848288149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50419f4c0 0x7fc50419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:47.666 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 >> 172.21.15.188:0/848288149 conn(0x7fc5040fe0a0 msgr2=0x7fc5040ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:47.666 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 shutdown_connections 2024-04-13T03:24:47.666 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:47.665+0000 7fc509c5c700 1 -- 172.21.15.188:0/848288149 wait complete. 2024-04-13T03:24:47.666 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 3 2024-04-13T03:24:48.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:48.197 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:48.197 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:48.197 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:48 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/848288149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-13T03:24:48.197 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:48.220 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-13T03:24:48.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph config generate-minimal-conf 2024-04-13T03:24:48.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:48.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:48 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/848288149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-13T03:24:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:48.652 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:24:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mon.c calling monitor election 2024-04-13T03:24:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mon.a calling monitor election 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: monmap epoch 3 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:49.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: last_changed 2024-04-13T03:24:41.998298+0000 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: min_mon_release 19 (squid) 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: election_strategy: 1 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: fsmap 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mgrmap e13: x(active, since 67s) 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: overall HEALTH_OK 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:49.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/848288149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-13T03:24:49.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:50.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: mon.b calling monitor election 2024-04-13T03:24:50.211 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-13T03:24:50.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-13T03:24:50.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-13T03:24:50.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: mon.b calling monitor election 2024-04-13T03:24:50.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: mon.c calling monitor election 2024-04-13T03:24:50.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: mon.a calling monitor election 2024-04-13T03:24:50.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-13T03:24:50.212 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: monmap epoch 3 2024-04-13T03:24:50.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:50.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: last_changed 2024-04-13T03:24:41.998298+0000 2024-04-13T03:24:50.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:24:50.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: min_mon_release 19 (squid) 2024-04-13T03:24:50.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: election_strategy: 1 2024-04-13T03:24:50.213 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:24:50.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-13T03:24:50.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-13T03:24:50.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: fsmap 2024-04-13T03:24:50.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:50.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: mgrmap e13: x(active, since 69s) 2024-04-13T03:24:50.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: overall HEALTH_OK 2024-04-13T03:24:50.215 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:50.215 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:50.215 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.215 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.215 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.215 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.216 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:49 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mon.b calling monitor election 2024-04-13T03:24:50.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-13T03:24:50.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-13T03:24:50.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-13T03:24:50.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mon.b calling monitor election 2024-04-13T03:24:50.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mon.c calling monitor election 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mon.a calling monitor election 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: monmap epoch 3 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: last_changed 2024-04-13T03:24:41.998298+0000 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: min_mon_release 19 (squid) 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: election_strategy: 1 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-13T03:24:50.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: fsmap 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: mgrmap e13: x(active, since 69s) 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: overall HEALTH_OK 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:49 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: mon.b calling monitor election 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: mon.b calling monitor election 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: mon.c calling monitor election 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: mon.a calling monitor election 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: monmap epoch 3 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: last_changed 2024-04-13T03:24:41.998298+0000 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: created 2024-04-13T03:22:18.286080+0000 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: min_mon_release 19 (squid) 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: election_strategy: 1 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-13T03:24:50.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: 1: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.c 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: fsmap 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: osdmap e4: 0 total, 0 up, 0 in 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: mgrmap e13: x(active, since 69s) 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: overall HEALTH_OK 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:49 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.743+0000 7efe28600700 1 -- 172.21.15.16:0/4022979486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 msgr2=0x7efe200752c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.743+0000 7efe28600700 1 --2- 172.21.15.16:0/4022979486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 0x7efe200752c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efe14009910 tx=0x7efe14009c20 comp rx=0 tx=0).stop 2024-04-13T03:24:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.743+0000 7efe28600700 1 -- 172.21.15.16:0/4022979486 shutdown_connections 2024-04-13T03:24:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.743+0000 7efe28600700 1 --2- 172.21.15.16:0/4022979486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 0x7efe200752c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.743+0000 7efe28600700 1 -- 172.21.15.16:0/4022979486 >> 172.21.15.16:0/4022979486 conn(0x7efe200fdff0 msgr2=0x7efe20100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.744+0000 7efe28600700 1 -- 172.21.15.16:0/4022979486 shutdown_connections 2024-04-13T03:24:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.744+0000 7efe28600700 1 -- 172.21.15.16:0/4022979486 wait complete. 2024-04-13T03:24:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.745+0000 7efe28600700 1 Processor -- start 2024-04-13T03:24:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.745+0000 7efe28600700 1 -- start start 2024-04-13T03:24:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.745+0000 7efe28600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 0x7efe2019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.745+0000 7efe28600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe2019aa20 0x7efe2019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.745+0000 7efe28600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2019f2f0 0x7efe2019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.745+0000 7efe28600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe2010ed70 con 0x7efe20074ee0 2024-04-13T03:24:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.745+0000 7efe28600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efe2010ebf0 con 0x7efe2019f2f0 2024-04-13T03:24:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.745+0000 7efe28600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efe2010eef0 con 0x7efe2019aa20 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 0x7efe2019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 0x7efe2019a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49236/0 (socket says 172.21.15.16:49236) 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe2019aa20 0x7efe2019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25c00700 1 -- 172.21.15.16:0/840483495 learned_addr learned my addr 172.21.15.16:0/840483495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe26600700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2019f2f0 0x7efe2019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25c00700 1 -- 172.21.15.16:0/840483495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe2019aa20 msgr2=0x7efe2019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25c00700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe2019aa20 0x7efe2019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25c00700 1 -- 172.21.15.16:0/840483495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2019f2f0 msgr2=0x7efe2019f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25c00700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2019f2f0 0x7efe2019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25c00700 1 -- 172.21.15.16:0/840483495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe201a7060 con 0x7efe20074ee0 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.746+0000 7efe25200700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe2019aa20 0x7efe2019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:24:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.747+0000 7efe25c00700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 0x7efe2019a4e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe140092e0 tx=0x7efe14004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.747+0000 7efe1e000700 1 -- 172.21.15.16:0/840483495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14026070 con 0x7efe20074ee0 2024-04-13T03:24:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.747+0000 7efe1e000700 1 -- 172.21.15.16:0/840483495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe1400b750 con 0x7efe20074ee0 2024-04-13T03:24:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.747+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe201a7230 con 0x7efe20074ee0 2024-04-13T03:24:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.747+0000 7efe1e000700 1 -- 172.21.15.16:0/840483495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14003a00 con 0x7efe20074ee0 2024-04-13T03:24:50.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.747+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe201a7760 con 0x7efe20074ee0 2024-04-13T03:24:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.748+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe20064110 con 0x7efe20074ee0 2024-04-13T03:24:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.749+0000 7efe1e000700 1 -- 172.21.15.16:0/840483495 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efe14029070 con 0x7efe20074ee0 2024-04-13T03:24:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.749+0000 7efe1e000700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe08039ae0 0x7efe0803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:50.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.749+0000 7efe1e000700 1 -- 172.21.15.16:0/840483495 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efe1404d390 con 0x7efe20074ee0 2024-04-13T03:24:50.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.750+0000 7efe25200700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe08039ae0 0x7efe0803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:50.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.750+0000 7efe25200700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe08039ae0 0x7efe0803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009ec0 tx=0x7efe0c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:50.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.755+0000 7efe1e000700 1 -- 172.21.15.16:0/840483495 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe14013070 con 0x7efe20074ee0 2024-04-13T03:24:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.993+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7efe2019b960 con 0x7efe20074ee0 2024-04-13T03:24:50.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.993+0000 7efe1e000700 1 -- 172.21.15.16:0/840483495 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7efe1400ee50 con 0x7efe20074ee0 2024-04-13T03:24:50.994 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:50.994 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-04-13T03:24:50.994 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:24:50.994 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-04-13T03:24:50.994 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:50.994 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-13T03:24:50.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:50.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe08039ae0 msgr2=0x7efe0803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:50.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe08039ae0 0x7efe0803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009ec0 tx=0x7efe0c009210 comp rx=0 tx=0).stop 2024-04-13T03:24:50.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 msgr2=0x7efe2019a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:50.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 0x7efe2019a4e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe140092e0 tx=0x7efe14004fa0 comp rx=0 tx=0).stop 2024-04-13T03:24:50.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 shutdown_connections 2024-04-13T03:24:50.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe08039ae0 0x7efe0803bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:50.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2019f2f0 0x7efe2019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:50.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe20074ee0 0x7efe2019a4e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:50.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 --2- 172.21.15.16:0/840483495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe2019aa20 0x7efe2019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:50.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.997+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 >> 172.21.15.16:0/840483495 conn(0x7efe200fdff0 msgr2=0x7efe201068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:50.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.998+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 shutdown_connections 2024-04-13T03:24:50.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:24:50.998+0000 7efe28600700 1 -- 172.21.15.16:0/840483495 wait complete. 2024-04-13T03:24:51.275 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:51.276 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-13T03:24:51.277 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:24:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:24:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-13T03:24:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:51.805 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-13T03:24:51.806 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:24:51.806 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-04-13T03:24:51.885 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:24:51.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:52.009 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:51 smithi016 ceph-mon[31772]: Reconfiguring mon.a (unknown last config time)... 2024-04-13T03:24:52.009 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:51 smithi016 ceph-mon[31772]: Reconfiguring daemon mon.a on smithi016 2024-04-13T03:24:52.009 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:51 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/840483495' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:52.013 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:24:52.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.conf 2024-04-13T03:24:52.042 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:51 smithi119 ceph-mon[35835]: Reconfiguring mon.a (unknown last config time)... 2024-04-13T03:24:52.042 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:51 smithi119 ceph-mon[35835]: Reconfiguring daemon mon.a on smithi016 2024-04-13T03:24:52.043 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:51 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/840483495' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:52.047 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:24:52.047 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:52.117 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:24:52.117 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2024-04-13T03:24:52.152 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:24:52.152 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:24:52.218 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:51 smithi188 ceph-mon[36398]: Reconfiguring mon.a (unknown last config time)... 2024-04-13T03:24:52.218 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:51 smithi188 ceph-mon[36398]: Reconfiguring daemon mon.a on smithi016 2024-04-13T03:24:52.218 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:51 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/840483495' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:52.222 INFO:tasks.cephadm:Adding mgr.x on smithi016 2024-04-13T03:24:52.222 INFO:tasks.cephadm:Adding mgr.y on smithi119 2024-04-13T03:24:52.222 INFO:tasks.cephadm:Adding mgr.z on smithi188 2024-04-13T03:24:52.222 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch apply mgr '3;smithi016=x;smithi119=y;smithi188=z' 2024-04-13T03:24:52.503 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:24:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:52 smithi119 ceph-mon[35835]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:52 smithi016 ceph-mon[31772]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:53.361 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:52 smithi188 ceph-mon[36398]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:53.853 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.851+0000 7f3d72693700 1 -- 172.21.15.188:0/4282206175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d6c102640 msgr2=0x7f3d6c102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:53.853 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.851+0000 7f3d72693700 1 --2- 172.21.15.188:0/4282206175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d6c102640 0x7f3d6c102a60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009910 tx=0x7f3d5c009c20 comp rx=0 tx=0).stop 2024-04-13T03:24:53.853 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.851+0000 7f3d72693700 1 -- 172.21.15.188:0/4282206175 shutdown_connections 2024-04-13T03:24:53.853 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.851+0000 7f3d72693700 1 --2- 172.21.15.188:0/4282206175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d6c102640 0x7f3d6c102a60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:53.853 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.851+0000 7f3d72693700 1 -- 172.21.15.188:0/4282206175 >> 172.21.15.188:0/4282206175 conn(0x7f3d6c0fdbe0 msgr2=0x7f3d6c100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:53.853 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.851+0000 7f3d72693700 1 -- 172.21.15.188:0/4282206175 shutdown_connections 2024-04-13T03:24:53.853 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.851+0000 7f3d72693700 1 -- 172.21.15.188:0/4282206175 wait complete. 2024-04-13T03:24:53.853 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.851+0000 7f3d72693700 1 Processor -- start 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.851+0000 7f3d72693700 1 -- start start 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d72693700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d6c102640 0x7f3d6c197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d72693700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d6c1983a0 0x7f3d6c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d72693700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d6c198820 0x7f3d6c19f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d72693700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c108680 con 0x7f3d6c1983a0 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d72693700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c108500 con 0x7f3d6c198820 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d72693700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c108800 con 0x7f3d6c102640 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d6c102640 0x7f3d6c197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d6c102640 0x7f3d6c197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:38506/0 (socket says 172.21.15.188:38506) 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6bfff700 1 -- 172.21.15.188:0/547130077 learned_addr learned my addr 172.21.15.188:0/547130077 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d70c30700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d6c198820 0x7f3d6c19f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6b7fe700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d6c1983a0 0x7f3d6c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:53.854 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6bfff700 1 -- 172.21.15.188:0/547130077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d6c198820 msgr2=0x7f3d6c19f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6bfff700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d6c198820 0x7f3d6c19f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6bfff700 1 -- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d6c1983a0 msgr2=0x7f3d6c19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6bfff700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d6c1983a0 0x7f3d6c19d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6bfff700 1 -- 172.21.15.188:0/547130077 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c1a0270 con 0x7f3d6c102640 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d70c30700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d6c198820 0x7f3d6c19f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.852+0000 7f3d6bfff700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d6c102640 0x7f3d6c197e60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c000c00 tx=0x7f3d5c0050b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.853+0000 7f3d697fa700 1 -- 172.21.15.188:0/547130077 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c027070 con 0x7f3d6c102640 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.853+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6c1a0500 con 0x7f3d6c102640 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.853+0000 7f3d697fa700 1 -- 172.21.15.188:0/547130077 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d5c022470 con 0x7f3d6c102640 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.853+0000 7f3d697fa700 1 -- 172.21.15.188:0/547130077 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c00f6c0 con 0x7f3d6c102640 2024-04-13T03:24:53.855 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.853+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6c1a0ab0 con 0x7f3d6c102640 2024-04-13T03:24:53.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.853+0000 7f3d697fa700 1 -- 172.21.15.188:0/547130077 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3d5c02a070 con 0x7f3d6c102640 2024-04-13T03:24:53.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.854+0000 7f3d697fa700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3d54039ae0 0x7f3d5403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:24:53.856 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.854+0000 7f3d697fa700 1 -- 172.21.15.188:0/547130077 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3d5c04e990 con 0x7f3d6c102640 2024-04-13T03:24:53.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.854+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d6c02dfa0 con 0x7f3d6c102640 2024-04-13T03:24:53.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.854+0000 7f3d6b7fe700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3d54039ae0 0x7f3d5403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:24:53.857 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.855+0000 7f3d6b7fe700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3d54039ae0 0x7f3d5403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d600078e0 tx=0x7f3d60008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:24:53.859 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:53.859+0000 7f3d697fa700 1 -- 172.21.15.188:0/547130077 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d5c013070 con 0x7f3d6c102640 2024-04-13T03:24:54.064 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.065+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi119=y;smithi188=z", "target": ["mon-mgr", ""]}) -- 0x7f3d6c062ef0 con 0x7f3d54039ae0 2024-04-13T03:24:54.081 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.081+0000 7f3d697fa700 1 -- 172.21.15.188:0/547130077 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3d6c062ef0 con 0x7f3d54039ae0 2024-04-13T03:24:54.081 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mgr update... 2024-04-13T03:24:54.083 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.083+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3d54039ae0 msgr2=0x7f3d5403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.083+0000 7f3d72693700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3d54039ae0 0x7f3d5403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d600078e0 tx=0x7f3d60008040 comp rx=0 tx=0).stop 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.083+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d6c102640 msgr2=0x7f3d6c197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.083+0000 7f3d72693700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d6c102640 0x7f3d6c197e60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c000c00 tx=0x7f3d5c0050b0 comp rx=0 tx=0).stop 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.083+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 shutdown_connections 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.083+0000 7f3d72693700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3d54039ae0 0x7f3d5403bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.083+0000 7f3d72693700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d6c198820 0x7f3d6c19f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.083+0000 7f3d72693700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3d6c102640 0x7f3d6c197e60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.083+0000 7f3d72693700 1 --2- 172.21.15.188:0/547130077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d6c1983a0 0x7f3d6c19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.084+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 >> 172.21.15.188:0/547130077 conn(0x7f3d6c0fdbe0 msgr2=0x7f3d6c0fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.084+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 shutdown_connections 2024-04-13T03:24:54.084 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:24:54.084+0000 7f3d72693700 1 -- 172.21.15.188:0/547130077 wait complete. 2024-04-13T03:24:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:54 smithi119 ceph-mon[35835]: Reconfiguring mon.b (monmap changed)... 2024-04-13T03:24:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-13T03:24:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:54 smithi119 ceph-mon[35835]: Reconfiguring daemon mon.b on smithi119 2024-04-13T03:24:54.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:54.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:54.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:54 smithi016 ceph-mon[31772]: Reconfiguring mon.b (monmap changed)... 2024-04-13T03:24:54.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:54.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-13T03:24:54.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:54.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:54 smithi016 ceph-mon[31772]: Reconfiguring daemon mon.b on smithi119 2024-04-13T03:24:54.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:54.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:54.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:54 smithi188 ceph-mon[36398]: Reconfiguring mon.b (monmap changed)... 2024-04-13T03:24:54.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:54.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-13T03:24:54.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:54.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:54 smithi188 ceph-mon[36398]: Reconfiguring daemon mon.b on smithi119 2024-04-13T03:24:54.686 DEBUG:teuthology.orchestra.run.smithi119:mgr.y> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.y.service 2024-04-13T03:24:54.689 DEBUG:teuthology.orchestra.run.smithi188:mgr.z> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.z.service 2024-04-13T03:24:54.691 INFO:tasks.cephadm:Deploying OSDs... 2024-04-13T03:24:54.692 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:24:54.692 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-04-13T03:24:54.753 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-13T03:24:54.753 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-04-13T03:24:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-13T03:24:54.812 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:54.812 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-13T03:24:54.812 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:54.813 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:54.813 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-13 03:23:52.688117754 +0000 2024-04-13T03:24:54.813 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-13 03:18:19.793410389 +0000 2024-04-13T03:24:54.813 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-13 03:18:19.793410389 +0000 2024-04-13T03:24:54.813 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-13 03:18:19.793410389 +0000 2024-04-13T03:24:54.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-13T03:24:54.906 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-13T03:24:54.907 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-13T03:24:54.907 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000273395 s, 1.9 MB/s 2024-04-13T03:24:54.911 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-13T03:24:54.968 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-04-13T03:24:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-13T03:24:55.018 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:55.018 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-13T03:24:55.018 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:55.018 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:55.018 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-13 03:23:52.747116844 +0000 2024-04-13T03:24:55.018 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-13 03:18:20.405401156 +0000 2024-04-13T03:24:55.019 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-13 03:18:20.405401156 +0000 2024-04-13T03:24:55.019 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-13 03:18:20.405401156 +0000 2024-04-13T03:24:55.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-13T03:24:55.106 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-13T03:24:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-13T03:24:55.107 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000273741 s, 1.9 MB/s 2024-04-13T03:24:55.111 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-13T03:24:55.170 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-04-13T03:24:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-13T03:24:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:55.223 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-04-13T03:24:55.223 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:55.223 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:55.223 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-13 03:23:52.800116028 +0000 2024-04-13T03:24:55.223 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-13 03:18:21.007392074 +0000 2024-04-13T03:24:55.223 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-13 03:18:21.007392074 +0000 2024-04-13T03:24:55.223 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-13 03:18:21.007392074 +0000 2024-04-13T03:24:55.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-13T03:24:55.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:55 smithi016 ceph-mon[31772]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:55.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:55 smithi016 ceph-mon[31772]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi119=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:55.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:55 smithi016 ceph-mon[31772]: Saving service mgr spec with placement smithi016=x;smithi119=y;smithi188=z;count:3 2024-04-13T03:24:55.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:55.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:55.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:55.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:55.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-13T03:24:55.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-13T03:24:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-13T03:24:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000279396 s, 1.8 MB/s 2024-04-13T03:24:55.325 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-13T03:24:55.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:55 smithi188 ceph-mon[36398]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:55.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:55 smithi188 ceph-mon[36398]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi119=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:55.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:55 smithi188 ceph-mon[36398]: Saving service mgr spec with placement smithi016=x;smithi119=y;smithi188=z;count:3 2024-04-13T03:24:55.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:55 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:55.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:55 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:55.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:55 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:55.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:55 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:55.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:55 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-13T03:24:55.371 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:55 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:55.385 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-04-13T03:24:55.439 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-13T03:24:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:55.440 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-13T03:24:55.440 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:55.440 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:55.440 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-13 03:23:52.856115165 +0000 2024-04-13T03:24:55.440 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-13 03:18:21.665382147 +0000 2024-04-13T03:24:55.440 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-13 03:18:21.665382147 +0000 2024-04-13T03:24:55.440 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-13 03:18:21.665382147 +0000 2024-04-13T03:24:55.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-13T03:24:55.535 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-13T03:24:55.535 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-13T03:24:55.535 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00027075 s, 1.9 MB/s 2024-04-13T03:24:55.539 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-13T03:24:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:55 smithi119 ceph-mon[35835]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:55 smithi119 ceph-mon[35835]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi016=x;smithi119=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:24:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:55 smithi119 ceph-mon[35835]: Saving service mgr spec with placement smithi016=x;smithi119=y;smithi188=z;count:3 2024-04-13T03:24:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:55 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:55 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:55 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:55 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-13T03:24:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:55 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-13T03:24:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:55 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:55.596 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-13T03:24:55.596 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-04-13T03:24:55.615 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-13T03:24:55.615 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-04-13T03:24:55.673 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-13T03:24:55.673 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:55.673 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-04-13T03:24:55.673 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:55.673 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:55.673 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-13 03:24:41.403188205 +0000 2024-04-13T03:24:55.673 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-13 03:16:51.697311023 +0000 2024-04-13T03:24:55.673 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-13 03:16:51.697311023 +0000 2024-04-13T03:24:55.673 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-13 03:16:51.697311023 +0000 2024-04-13T03:24:55.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-13T03:24:55.742 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-13T03:24:55.743 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-13T03:24:55.743 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000208383 s, 2.5 MB/s 2024-04-13T03:24:55.744 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-13T03:24:55.801 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-04-13T03:24:55.858 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-13T03:24:55.858 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:55.858 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-04-13T03:24:55.858 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:55.859 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:55.859 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-13 03:24:41.404188180 +0000 2024-04-13T03:24:55.859 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-13 03:16:52.037302414 +0000 2024-04-13T03:24:55.859 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-13 03:16:52.037302414 +0000 2024-04-13T03:24:55.859 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-13 03:16:52.037302414 +0000 2024-04-13T03:24:55.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-13T03:24:55.926 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-13T03:24:55.926 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-13T03:24:55.927 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000210341 s, 2.4 MB/s 2024-04-13T03:24:55.928 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-13T03:24:55.985 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-04-13T03:24:56.040 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-13T03:24:56.041 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:56.041 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-13T03:24:56.041 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:56.041 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:56.041 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-13 03:24:41.404188180 +0000 2024-04-13T03:24:56.041 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-13 03:16:52.374293881 +0000 2024-04-13T03:24:56.041 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-13 03:16:52.374293881 +0000 2024-04-13T03:24:56.041 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-13 03:16:52.374293881 +0000 2024-04-13T03:24:56.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-13T03:24:56.110 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-13T03:24:56.110 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-13T03:24:56.110 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000174696 s, 2.9 MB/s 2024-04-13T03:24:56.110 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-13T03:24:56.166 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-04-13T03:24:56.221 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-13T03:24:56.221 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:56.221 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-04-13T03:24:56.221 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:56.221 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:56.221 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-13 03:24:41.404188180 +0000 2024-04-13T03:24:56.221 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-13 03:16:52.698285678 +0000 2024-04-13T03:24:56.221 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-13 03:16:52.698285678 +0000 2024-04-13T03:24:56.221 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-13 03:16:52.698285678 +0000 2024-04-13T03:24:56.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-13T03:24:56.287 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-13T03:24:56.287 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-13T03:24:56.287 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000216433 s, 2.4 MB/s 2024-04-13T03:24:56.288 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-13T03:24:56.344 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-13T03:24:56.344 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-04-13T03:24:56.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: Reconfiguring mon.c (monmap changed)... 2024-04-13T03:24:56.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: Reconfiguring daemon mon.c on smithi188 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-13T03:24:56.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:56.359 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-13T03:24:56.359 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: Reconfiguring mon.c (monmap changed)... 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: Reconfiguring daemon mon.c on smithi188 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-13T03:24:56.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:56.388 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-13T03:24:56.388 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:56.388 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-04-13T03:24:56.388 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:56.388 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:56.388 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-13 03:24:32.300428864 +0000 2024-04-13T03:24:56.388 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-13 03:17:00.914454176 +0000 2024-04-13T03:24:56.388 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-13 03:17:00.914454176 +0000 2024-04-13T03:24:56.388 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-13 03:17:00.914454176 +0000 2024-04-13T03:24:56.389 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-13T03:24:56.456 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-13T03:24:56.456 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-13T03:24:56.456 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000255294 s, 2.0 MB/s 2024-04-13T03:24:56.457 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-13T03:24:56.513 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-04-13T03:24:56.568 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-13T03:24:56.568 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:56.568 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-04-13T03:24:56.569 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:56.569 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:56.569 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-13 03:24:32.301428845 +0000 2024-04-13T03:24:56.569 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-13 03:17:01.258447678 +0000 2024-04-13T03:24:56.569 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-13 03:17:01.258447678 +0000 2024-04-13T03:24:56.569 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-13 03:17:01.258447678 +0000 2024-04-13T03:24:56.569 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: Reconfiguring mon.c (monmap changed)... 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: Reconfiguring daemon mon.c on smithi188 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-13T03:24:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-13T03:24:56.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:56.633 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-13T03:24:56.634 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-13T03:24:56.634 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000216778 s, 2.4 MB/s 2024-04-13T03:24:56.635 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-13T03:24:56.691 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-04-13T03:24:56.746 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-13T03:24:56.746 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:56.746 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-13T03:24:56.746 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:56.746 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:56.747 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-13 03:24:32.301428845 +0000 2024-04-13T03:24:56.747 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-13 03:17:01.600441218 +0000 2024-04-13T03:24:56.747 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-13 03:17:01.600441218 +0000 2024-04-13T03:24:56.747 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-13 03:17:01.600441218 +0000 2024-04-13T03:24:56.747 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-13T03:24:56.812 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-13T03:24:56.812 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-13T03:24:56.812 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000177571 s, 2.9 MB/s 2024-04-13T03:24:56.813 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-13T03:24:56.869 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-04-13T03:24:56.923 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-13T03:24:56.923 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-13T03:24:56.923 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-04-13T03:24:56.923 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:24:56.923 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-13T03:24:56.923 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-13 03:24:32.302428827 +0000 2024-04-13T03:24:56.923 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-13 03:17:01.948434646 +0000 2024-04-13T03:24:56.923 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-13 03:17:01.948434646 +0000 2024-04-13T03:24:56.923 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-13 03:17:01.948434646 +0000 2024-04-13T03:24:56.923 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-13T03:24:56.987 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-13T03:24:56.988 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-13T03:24:56.988 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000200423 s, 2.6 MB/s 2024-04-13T03:24:56.989 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-13T03:24:57.045 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-04-13T03:24:57.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-13T03:24:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:57 smithi016 ceph-mon[31772]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:57 smithi016 ceph-mon[31772]: Deploying daemon mgr.z on smithi188 2024-04-13T03:24:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:57 smithi188 ceph-mon[36398]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:57 smithi188 ceph-mon[36398]: Deploying daemon mgr.z on smithi188 2024-04-13T03:24:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:24:57.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:57 smithi119 ceph-mon[35835]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:57.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:57 smithi119 ceph-mon[35835]: Deploying daemon mgr.z on smithi188 2024-04-13T03:24:58.122 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:57 smithi188 systemd[1]: Starting Ceph mgr.z for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:24:58.462 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:58 smithi188 podman[38247]: 2024-04-13 03:24:58.11382478 +0000 UTC m=+0.017754987 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:24:58.462 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:58 smithi188 podman[38247]: 2024-04-13 03:24:58.232454234 +0000 UTC m=+0.136384478 container create 6df30516e333a4c4372033ca7f395464d5552c65be2760891ad0d35e80c68cde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-04-13T03:24:58.463 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:58 smithi188 podman[38247]: 2024-04-13 03:24:58.459856048 +0000 UTC m=+0.363786250 container init 6df30516e333a4c4372033ca7f395464d5552c65be2760891ad0d35e80c68cde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-13T03:24:58.463 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:58 smithi188 podman[38247]: 2024-04-13 03:24:58.46237528 +0000 UTC m=+0.366305481 container start 6df30516e333a4c4372033ca7f395464d5552c65be2760891ad0d35e80c68cde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-13T03:24:58.872 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:58 smithi188 ceph-mgr[38266]: -- 172.21.15.188:0/1557902047 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629c8e905a0 con 0x5629c8e64c00 2024-04-13T03:24:58.872 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:58 smithi188 bash[38247]: 6df30516e333a4c4372033ca7f395464d5552c65be2760891ad0d35e80c68cde 2024-04-13T03:24:58.872 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:58 smithi188 systemd[1]: Started Ceph mgr.z for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:24:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:59 smithi016 ceph-mon[31772]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-13T03:24:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-13T03:24:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-13T03:24:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:24:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:59.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:59 smithi119 ceph-mon[35835]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:59.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:59 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:59.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:59 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:59.361 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:59 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:59.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:59 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-13T03:24:59.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:59 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-13T03:24:59.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:59 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-13T03:24:59.362 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:24:59 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:59.372 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:24:59.003+0000 7fc4e8d14200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T03:24:59.372 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:24:59.095+0000 7fc4e8d14200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T03:24:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-mon[36398]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:24:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:24:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-13T03:24:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-13T03:24:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-13T03:24:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:59.970 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:24:59.738+0000 7fc4e8d14200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T03:24:59.971 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:24:59.829+0000 7fc4e8d14200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T03:24:59.971 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:24:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:24:59.969+0000 7fc4e8d14200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T03:25:00.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:00 smithi016 ceph-mon[31772]: Deploying daemon mgr.y on smithi119 2024-04-13T03:25:00.367 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:00 smithi119 ceph-mon[35835]: Deploying daemon mgr.y on smithi119 2024-04-13T03:25:00.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:00 smithi188 ceph-mon[36398]: Deploying daemon mgr.y on smithi119 2024-04-13T03:25:00.373 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:00.064+0000 7fc4e8d14200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T03:25:00.373 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:00.147+0000 7fc4e8d14200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T03:25:00.649 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-13T03:25:00.649 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-13T03:25:00.649 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-13T03:25:00.649 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-13T03:25:00.649 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0387497 s, 271 MB/s 2024-04-13T03:25:00.649 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 2024-04-13T03:25:00.650 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-13T03:25:00.693 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:00 smithi119 systemd[1]: Starting Ceph mgr.y for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:25:00.698 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:00.411+0000 7fc4e8d14200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T03:25:00.698 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:00.519+0000 7fc4e8d14200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T03:25:00.698 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:00.604+0000 7fc4e8d14200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T03:25:00.698 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:00.697+0000 7fc4e8d14200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T03:25:01.011 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:00 smithi119 podman[37217]: 2024-04-13 03:25:00.660421987 +0000 UTC m=+0.017925854 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:25:01.012 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:00 smithi119 podman[37217]: 2024-04-13 03:25:00.801159583 +0000 UTC m=+0.158663431 container create 9069651ddd42eb3f663903251e02570c451dcf5ecf8726968eb0453cfa57ce99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-13T03:25:01.012 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:01 smithi119 podman[37217]: 2024-04-13 03:25:01.011814208 +0000 UTC m=+0.369318062 container init 9069651ddd42eb3f663903251e02570c451dcf5ecf8726968eb0453cfa57ce99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:25:01.101 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:00.874+0000 7fc4e8d14200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T03:25:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:01 smithi119 ceph-mon[35835]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:01.322 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:01 smithi119 podman[37217]: 2024-04-13 03:25:01.014308323 +0000 UTC m=+0.371812174 container start 9069651ddd42eb3f663903251e02570c451dcf5ecf8726968eb0453cfa57ce99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409) 2024-04-13T03:25:01.322 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:01 smithi119 ceph-mgr[37237]: -- 172.21.15.119:0/3663643384 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd23b0c5a0 con 0x55bd23ae0800 2024-04-13T03:25:01.322 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:01 smithi119 bash[37217]: 9069651ddd42eb3f663903251e02570c451dcf5ecf8726968eb0453cfa57ce99 2024-04-13T03:25:01.322 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:01 smithi119 systemd[1]: Started Ceph mgr.y for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:25:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:01 smithi016 ceph-mon[31772]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:01 smithi188 ceph-mon[36398]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:01.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-04-13T03:25:01.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:01 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:01.560+0000 7fb12dbde200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T03:25:01.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:01 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:01.658+0000 7fb12dbde200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T03:25:01.952 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:25:02.122 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:01 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:01.862+0000 7fc4e8d14200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T03:25:02.123 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:01 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:01.996+0000 7fc4e8d14200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T03:25:02.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:02 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:02 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:02 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:02 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:02 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:02.444 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:02.351+0000 7fc4e8d14200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T03:25:02.444 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:02.443+0000 7fc4e8d14200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T03:25:02.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:02 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:02 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:02 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:02 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:02 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:02.468 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.468 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.468 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.468 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:02.468 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:02.468 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:02.331+0000 7fb12dbde200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T03:25:02.468 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:02.427+0000 7fb12dbde200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T03:25:02.752 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:02.573+0000 7fb12dbde200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T03:25:02.752 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:02.669+0000 7fb12dbde200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T03:25:02.752 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:02 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:02.751+0000 7fb12dbde200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T03:25:02.838 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:02.647+0000 7fc4e8d14200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:25:02.839 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:02.837+0000 7fc4e8d14200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T03:25:03.316 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:03.025+0000 7fb12dbde200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T03:25:03.317 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:03.137+0000 7fb12dbde200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T03:25:03.317 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:03.221+0000 7fb12dbde200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T03:25:03.317 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:03.315+0000 7fb12dbde200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T03:25:03.317 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:03.317 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.317 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.317 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.317 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.318 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:03.318 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:03.318 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.318 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-13T03:25:03.318 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-13T03:25:03.318 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-13T03:25:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-13T03:25:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:03.387 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:03.124+0000 7fc4e8d14200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:25:03.387 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:03.386+0000 7fc4e8d14200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T03:25:03.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:03.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.387 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:03.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:03.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:03.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-13T03:25:03.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-13T03:25:03.388 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:03.571 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:03 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:03.489+0000 7fb12dbde200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T03:25:03.872 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:03 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:03.619+0000 7fc4e8d14200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T03:25:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.389+0000 7fdc0ce00700 1 -- 172.21.15.16:0/1812955841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc0810acf0 msgr2=0x7fdc081115a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.389+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/1812955841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc0810acf0 0x7fdc081115a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000b0d0 tx=0x7fdc0000b3e0 comp rx=0 tx=0).stop 2024-04-13T03:25:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.389+0000 7fdc0ce00700 1 -- 172.21.15.16:0/1812955841 shutdown_connections 2024-04-13T03:25:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.389+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/1812955841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc0810acf0 0x7fdc081115a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.389+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/1812955841 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc08109150 0x7fdc08109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.389+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/1812955841 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0810a350 0x7fdc0810a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.389+0000 7fdc0ce00700 1 -- 172.21.15.16:0/1812955841 >> 172.21.15.16:0/1812955841 conn(0x7fdc08078120 msgr2=0x7fdc0807a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:25:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.389+0000 7fdc0ce00700 1 -- 172.21.15.16:0/1812955841 shutdown_connections 2024-04-13T03:25:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.389+0000 7fdc0ce00700 1 -- 172.21.15.16:0/1812955841 wait complete. 2024-04-13T03:25:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.391+0000 7fdc0ce00700 1 Processor -- start 2024-04-13T03:25:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.391+0000 7fdc0ce00700 1 -- start start 2024-04-13T03:25:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.391+0000 7fdc0ce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc08109150 0x7fdc0819f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.391+0000 7fdc0ce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc0810a350 0x7fdc0819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.391+0000 7fdc0ce00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0810acf0 0x7fdc081a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.391+0000 7fdc0ce00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08114d90 con 0x7fdc0810a350 2024-04-13T03:25:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.391+0000 7fdc0ce00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08114c10 con 0x7fdc08109150 2024-04-13T03:25:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.391+0000 7fdc0ce00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdc08114f10 con 0x7fdc0810acf0 2024-04-13T03:25:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdc05e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc08109150 0x7fdc0819f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdc05e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc08109150 0x7fdc0819f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:35614/0 (socket says 172.21.15.16:35614) 2024-04-13T03:25:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdc05e00700 1 -- 172.21.15.16:0/4226929329 learned_addr learned my addr 172.21.15.16:0/4226929329 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:25:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdbffe00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc0810a350 0x7fdc0819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdc06800700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0810acf0 0x7fdc081a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdc06800700 1 -- 172.21.15.16:0/4226929329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc08109150 msgr2=0x7fdc0819f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdc06800700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc08109150 0x7fdc0819f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdc06800700 1 -- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc0810a350 msgr2=0x7fdc0819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdc06800700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc0810a350 0x7fdc0819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.392+0000 7fdc06800700 1 -- 172.21.15.16:0/4226929329 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc081a6f40 con 0x7fdc0810acf0 2024-04-13T03:25:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.393+0000 7fdc06800700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0810acf0 0x7fdc081a68a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdc00009fd0 tx=0x7fdc000079a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:25:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.393+0000 7fdbfea00700 1 -- 172.21.15.16:0/4226929329 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc00021070 con 0x7fdc0810acf0 2024-04-13T03:25:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.393+0000 7fdbfea00700 1 -- 172.21.15.16:0/4226929329 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0001db30 con 0x7fdc0810acf0 2024-04-13T03:25:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.393+0000 7fdbfea00700 1 -- 172.21.15.16:0/4226929329 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0001bba0 con 0x7fdc0810acf0 2024-04-13T03:25:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.393+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc081a71d0 con 0x7fdc0810acf0 2024-04-13T03:25:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.393+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc081a7630 con 0x7fdc0810acf0 2024-04-13T03:25:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.394+0000 7fdbfea00700 1 -- 172.21.15.16:0/4226929329 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fdc00024070 con 0x7fdc0810acf0 2024-04-13T03:25:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.394+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc080697a0 con 0x7fdc0810acf0 2024-04-13T03:25:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.395+0000 7fdbfea00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdbec039890 0x7fdbec03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.395+0000 7fdbfea00700 1 -- 172.21.15.16:0/4226929329 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdc0004da70 con 0x7fdc0810acf0 2024-04-13T03:25:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.395+0000 7fdc05e00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdbec039890 0x7fdbec03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.396+0000 7fdc05e00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdbec039890 0x7fdbec03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8007cf0 tx=0x7fdbf8007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:25:04.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.400+0000 7fdbfea00700 1 -- 172.21.15.16:0/4226929329 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc00009c60 con 0x7fdc0810acf0 2024-04-13T03:25:04.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:04 smithi016 ceph-mon[31772]: Reconfiguring mgr.x (unknown last config time)... 2024-04-13T03:25:04.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:04 smithi016 ceph-mon[31772]: Reconfiguring daemon mgr.x on smithi016 2024-04-13T03:25:04.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:04 smithi119 ceph-mon[35835]: Reconfiguring mgr.x (unknown last config time)... 2024-04-13T03:25:04.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:04 smithi119 ceph-mon[35835]: Reconfiguring daemon mgr.x on smithi016 2024-04-13T03:25:04.571 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:04.444+0000 7fb12dbde200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T03:25:04.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:04 smithi188 ceph-mon[36398]: Reconfiguring mgr.x (unknown last config time)... 2024-04-13T03:25:04.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:04 smithi188 ceph-mon[36398]: Reconfiguring daemon mgr.x on smithi016 2024-04-13T03:25:04.623 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:04.416+0000 7fc4e8d14200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T03:25:04.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:04.653+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fdc08062ef0 con 0x7fdbec039890 2024-04-13T03:25:04.900 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:04.573+0000 7fb12dbde200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T03:25:05.181 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:04.900+0000 7fb12dbde200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T03:25:05.181 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:04.987+0000 7fb12dbde200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T03:25:05.181 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:05 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:05.180+0000 7fb12dbde200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:25:05.242 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:04.986+0000 7fc4e8d14200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T03:25:05.243 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:05 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:05.083+0000 7fc4e8d14200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T03:25:05.571 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:05 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:05.361+0000 7fb12dbde200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T03:25:05.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:05 smithi119 ceph-mon[35835]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:05.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:05 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:25:05.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:05 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:25:05.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:05 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:05.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:05 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:05.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:05 smithi016 ceph-mon[31772]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:05.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:05 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:25:05.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:05 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:25:05.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:05 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:05.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:05 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:05.622 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:25:05 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:25:05.296+0000 7fc4e8d14200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T03:25:05.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:05 smithi188 ceph-mon[36398]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:05.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:05 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:25:05.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:05 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:25:05.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:05 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:05.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:05 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:05.899 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:05 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:05.643+0000 7fb12dbde200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:25:06.244 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:05 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:05.898+0000 7fb12dbde200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T03:25:06.245 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:06.133+0000 7fb12dbde200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T03:25:06.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:06.259+0000 7fdbfea00700 1 -- 172.21.15.16:0/4226929329 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7fdc0001dce0 con 0x7fdc0810acf0 2024-04-13T03:25:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:25:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: Standby manager daemon z started 2024-04-13T03:25:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-13T03:25:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:25:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-13T03:25:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:25:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: Standby manager daemon z started 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:06.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: Standby manager daemon z started 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.188:0/3030702956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:07.260 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:06 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:06.932+0000 7fb12dbde200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T03:25:07.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:07 smithi016 ceph-mon[31772]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:07.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:07 smithi016 ceph-mon[31772]: mgrmap e14: x(active, since 86s), standbys: z 2024-04-13T03:25:07.570 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-13T03:25:07.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:07 smithi119 ceph-mon[35835]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:07.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:07 smithi119 ceph-mon[35835]: mgrmap e14: x(active, since 86s), standbys: z 2024-04-13T03:25:07.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:07 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-13T03:25:07.571 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:07.502+0000 7fb12dbde200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T03:25:07.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:07 smithi188 ceph-mon[36398]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:07.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:07 smithi188 ceph-mon[36398]: mgrmap e14: x(active, since 86s), standbys: z 2024-04-13T03:25:07.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:07 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-13T03:25:08.071 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:07.597+0000 7fb12dbde200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T03:25:08.071 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:25:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:25:07.807+0000 7fb12dbde200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T03:25:08.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:08.279+0000 7fdbfea00700 1 -- 172.21.15.16:0/4226929329 <== mon.1 v2:172.21.15.188:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fdc00012c10 con 0x7fdc0810acf0 2024-04-13T03:25:08.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:08 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-13T03:25:08.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:08 smithi119 ceph-mon[35835]: Standby manager daemon y started 2024-04-13T03:25:08.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:08 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:25:08.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:08 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-13T03:25:08.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:08 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:25:08.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:08 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-13T03:25:08.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:08 smithi016 ceph-mon[31772]: Standby manager daemon y started 2024-04-13T03:25:08.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:08 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:25:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:08 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-13T03:25:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:08 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:25:08.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:08 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-13T03:25:08.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:08 smithi188 ceph-mon[36398]: Standby manager daemon y started 2024-04-13T03:25:08.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:08 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:25:08.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:08 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-13T03:25:08.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:08 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.119:0/3334184299' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:25:09.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:09 smithi119 ceph-mon[35835]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:09.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:09 smithi119 ceph-mon[35835]: mgrmap e15: x(active, since 88s), standbys: z, y 2024-04-13T03:25:09.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:09 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-13T03:25:09.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:09 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2207846668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69f3ffdc-4c06-47c2-92c0-1b7aab496918"}]: dispatch 2024-04-13T03:25:09.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:09 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2207846668' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69f3ffdc-4c06-47c2-92c0-1b7aab496918"}]': finished 2024-04-13T03:25:09.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:09 smithi119 ceph-mon[35835]: osdmap e5: 1 total, 0 up, 1 in 2024-04-13T03:25:09.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:09 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:09 smithi016 ceph-mon[31772]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:09 smithi016 ceph-mon[31772]: mgrmap e15: x(active, since 88s), standbys: z, y 2024-04-13T03:25:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:09 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-13T03:25:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:09 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2207846668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69f3ffdc-4c06-47c2-92c0-1b7aab496918"}]: dispatch 2024-04-13T03:25:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:09 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2207846668' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69f3ffdc-4c06-47c2-92c0-1b7aab496918"}]': finished 2024-04-13T03:25:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:09 smithi016 ceph-mon[31772]: osdmap e5: 1 total, 0 up, 1 in 2024-04-13T03:25:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:09 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:09.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:09 smithi188 ceph-mon[36398]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:09.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:09 smithi188 ceph-mon[36398]: mgrmap e15: x(active, since 88s), standbys: z, y 2024-04-13T03:25:09.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:09 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-13T03:25:09.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:09 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2207846668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69f3ffdc-4c06-47c2-92c0-1b7aab496918"}]: dispatch 2024-04-13T03:25:09.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:09 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2207846668' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69f3ffdc-4c06-47c2-92c0-1b7aab496918"}]': finished 2024-04-13T03:25:09.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:09 smithi188 ceph-mon[36398]: osdmap e5: 1 total, 0 up, 1 in 2024-04-13T03:25:09.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:09 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:10.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:10 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/709841788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:25:10.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:10 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/709841788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:25:10.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:10 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/709841788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:25:11.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:11 smithi119 ceph-mon[35835]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:11.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:11 smithi016 ceph-mon[31772]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:11.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:11 smithi188 ceph-mon[36398]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:13.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:13 smithi119 ceph-mon[35835]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:13.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:13 smithi016 ceph-mon[31772]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:13.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:13 smithi188 ceph-mon[36398]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:15 smithi119 ceph-mon[35835]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:15.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:15 smithi016 ceph-mon[31772]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:15.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:15 smithi188 ceph-mon[36398]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:17 smithi016 ceph-mon[31772]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:17 smithi119 ceph-mon[35835]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:17.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:17 smithi188 ceph-mon[36398]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:18.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-13T03:25:18.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:18.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-13T03:25:18.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:18.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-13T03:25:18.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:19 smithi119 ceph-mon[35835]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:19 smithi119 ceph-mon[35835]: Deploying daemon osd.0 on smithi016 2024-04-13T03:25:19.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:19 smithi016 ceph-mon[31772]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:19.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:19 smithi016 ceph-mon[31772]: Deploying daemon osd.0 on smithi016 2024-04-13T03:25:19.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:19 smithi188 ceph-mon[36398]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:19 smithi188 ceph-mon[36398]: Deploying daemon osd.0 on smithi016 2024-04-13T03:25:21.462 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:21 smithi016 ceph-mon[31772]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:21.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:21 smithi119 ceph-mon[35835]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:21.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:21 smithi188 ceph-mon[36398]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:23.470 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:23 smithi016 ceph-mon[31772]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:23.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:23 smithi188 ceph-mon[36398]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:23 smithi119 ceph-mon[35835]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:25.587 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:25 smithi016 ceph-mon[31772]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:25 smithi188 ceph-mon[36398]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:25.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:25 smithi119 ceph-mon[35835]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:27.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:26 smithi016 ceph-mon[31772]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:27.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:27.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:27.044 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:26 smithi119 ceph-mon[35835]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:26 smithi188 ceph-mon[36398]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:27.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:27.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:28.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:28 smithi016 ceph-mon[31772]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:29.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:28 smithi119 ceph-mon[35835]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:28 smithi188 ceph-mon[36398]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:30.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:30.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:30.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:30.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:30.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:30.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:30 smithi119 ceph-mon[35835]: from='osd.0 [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-13T03:25:30.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:30 smithi119 ceph-mon[35835]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-13T03:25:30.342 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:30.342 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:30.342 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:30.342 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:30.343 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:30.343 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:30 smithi016 ceph-mon[31772]: from='osd.0 [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-13T03:25:30.343 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:30 smithi016 ceph-mon[31772]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-13T03:25:30.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:30.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:30.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:30.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:30.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:30.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:30 smithi188 ceph-mon[36398]: from='osd.0 [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-13T03:25:30.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:30 smithi188 ceph-mon[36398]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-13T03:25:30.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.460+0000 7fdbfea00700 1 -- 172.21.15.16:0/4226929329 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdc08062ef0 con 0x7fdbec039890 2024-04-13T03:25:30.463 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-04-13T03:25:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdbec039890 msgr2=0x7fdbec03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdbec039890 0x7fdbec03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8007cf0 tx=0x7fdbf8007480 comp rx=0 tx=0).stop 2024-04-13T03:25:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0810acf0 msgr2=0x7fdc081a68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0810acf0 0x7fdc081a68a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdc00009fd0 tx=0x7fdc000079a0 comp rx=0 tx=0).stop 2024-04-13T03:25:30.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 shutdown_connections 2024-04-13T03:25:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdbec039890 0x7fdbec03bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0810acf0 0x7fdc081a68a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc08109150 0x7fdc0819f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 --2- 172.21.15.16:0/4226929329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc0810a350 0x7fdc0819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.464+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 >> 172.21.15.16:0/4226929329 conn(0x7fdc08078120 msgr2=0x7fdc0810d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:25:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.465+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 shutdown_connections 2024-04-13T03:25:30.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:25:30.465+0000 7fdc0ce00700 1 -- 172.21.15.16:0/4226929329 wait complete. 2024-04-13T03:25:31.240 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.0.service 2024-04-13T03:25:31.245 INFO:tasks.cephadm:Deploying osd.1 on smithi119 with /dev/vg_nvme/lv_4... 2024-04-13T03:25:31.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-13T03:25:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-mon[31772]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-mon[31772]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-13T03:25:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-mon[31772]: osdmap e6: 1 total, 0 up, 1 in 2024-04-13T03:25:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-mon[31772]: from='osd.0 [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:25:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-mon[31772]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:25:31.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:31.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:31.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:31 smithi188 ceph-mon[36398]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:31 smithi188 ceph-mon[36398]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-13T03:25:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:31 smithi188 ceph-mon[36398]: osdmap e6: 1 total, 0 up, 1 in 2024-04-13T03:25:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:31 smithi188 ceph-mon[36398]: from='osd.0 [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:25:31.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:31 smithi188 ceph-mon[36398]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:25:31.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:31.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:31.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:31.488 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:25:31.540 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:31 smithi119 ceph-mon[35835]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:31.540 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:31 smithi119 ceph-mon[35835]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-13T03:25:31.540 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:31 smithi119 ceph-mon[35835]: osdmap e6: 1 total, 0 up, 1 in 2024-04-13T03:25:31.540 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:31.540 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:31 smithi119 ceph-mon[35835]: from='osd.0 [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:25:31.540 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:31 smithi119 ceph-mon[35835]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:25:31.541 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:31.541 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:31.541 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:31.855 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0[43835]: 2024-04-13T03:25:31.688+0000 7f3449200700 -1 osd.0 0 waiting for initial osdmap 2024-04-13T03:25:31.855 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:25:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0[43835]: 2024-04-13T03:25:31.706+0000 7f343f000700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T03:25:32.220 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:32 smithi016 ceph-mon[31772]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:25:32.220 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:32 smithi016 ceph-mon[31772]: osdmap e7: 1 total, 0 up, 1 in 2024-04-13T03:25:32.221 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:32 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:32.221 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:32 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:32.221 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:32 smithi016 ceph-mon[31772]: from='osd.0 ' entity='osd.0' 2024-04-13T03:25:32.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:32 smithi119 ceph-mon[35835]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:25:32.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:32 smithi119 ceph-mon[35835]: osdmap e7: 1 total, 0 up, 1 in 2024-04-13T03:25:32.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:32 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:32.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:32 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:32.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:32 smithi119 ceph-mon[35835]: from='osd.0 ' entity='osd.0' 2024-04-13T03:25:32.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:32 smithi188 ceph-mon[36398]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:25:32.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:32 smithi188 ceph-mon[36398]: osdmap e7: 1 total, 0 up, 1 in 2024-04-13T03:25:32.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:32 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:32.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:32 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:32.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:32 smithi188 ceph-mon[36398]: from='osd.0 ' entity='osd.0' 2024-04-13T03:25:33.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:33 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:25:33.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:33 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:25:33.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:33 smithi016 ceph-mon[31772]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:33.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:33 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:33.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:33 smithi016 ceph-mon[31772]: osd.0 [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] boot 2024-04-13T03:25:33.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:33 smithi016 ceph-mon[31772]: osdmap e8: 1 total, 1 up, 1 in 2024-04-13T03:25:33.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:33 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:33.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:33 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:25:33.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:33 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:25:33.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:33 smithi119 ceph-mon[35835]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:33.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:33 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:33.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:33 smithi119 ceph-mon[35835]: osd.0 [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] boot 2024-04-13T03:25:33.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:33 smithi119 ceph-mon[35835]: osdmap e8: 1 total, 1 up, 1 in 2024-04-13T03:25:33.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:33 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:33 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:25:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:33 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:25:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:33 smithi188 ceph-mon[36398]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-13T03:25:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:33 smithi188 ceph-mon[36398]: osd.0 [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] boot 2024-04-13T03:25:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:33 smithi188 ceph-mon[36398]: osdmap e8: 1 total, 1 up, 1 in 2024-04-13T03:25:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:25:33.430 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-13T03:25:33.430 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-13T03:25:33.430 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-04-13T03:25:33.430 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-04-13T03:25:33.430 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112565 s, 932 MB/s 2024-04-13T03:25:33.430 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-04-13T03:25:34.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_4 2024-04-13T03:25:34.251 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:25:35.015 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:34 smithi119 ceph-mon[35835]: osdmap e9: 1 total, 1 up, 1 in 2024-04-13T03:25:35.015 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:34 smithi119 ceph-mon[35835]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:35.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:34 smithi016 ceph-mon[31772]: osdmap e9: 1 total, 1 up, 1 in 2024-04-13T03:25:35.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:34 smithi016 ceph-mon[31772]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:34 smithi188 ceph-mon[36398]: osdmap e9: 1 total, 1 up, 1 in 2024-04-13T03:25:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:34 smithi188 ceph-mon[36398]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.704+0000 7f576cdd7700 1 -- 172.21.15.119:0/3455676267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 msgr2=0x7f5768109e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.704+0000 7f576cdd7700 1 --2- 172.21.15.119:0/3455676267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 0x7f5768109e80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f576000b0d0 tx=0x7f576000b3e0 comp rx=0 tx=0).stop 2024-04-13T03:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.705+0000 7f576cdd7700 1 -- 172.21.15.119:0/3455676267 shutdown_connections 2024-04-13T03:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.705+0000 7f576cdd7700 1 --2- 172.21.15.119:0/3455676267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 0x7f5768109e80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.705+0000 7f576cdd7700 1 --2- 172.21.15.119:0/3455676267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5768101a30 0x7f5768101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.705+0000 7f576cdd7700 1 --2- 172.21.15.119:0/3455676267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5768102c30 0x7f5768103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.705+0000 7f576cdd7700 1 -- 172.21.15.119:0/3455676267 >> 172.21.15.119:0/3455676267 conn(0x7f57680fd010 msgr2=0x7f57680ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.705+0000 7f576cdd7700 1 -- 172.21.15.119:0/3455676267 shutdown_connections 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.705+0000 7f576cdd7700 1 -- 172.21.15.119:0/3455676267 wait complete. 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f576cdd7700 1 Processor -- start 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f576cdd7700 1 -- start start 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f576cdd7700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5768101a30 0x7f5768197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f576cdd7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5768102c30 0x7f5768198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f576cdd7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 0x7f576819f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f576cdd7700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f576810d710 con 0x7f5768101a30 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f576cdd7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f576810d590 con 0x7f57681035d0 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f576cdd7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f576810d890 con 0x7f5768102c30 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5766d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 0x7f576819f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5766d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 0x7f576819f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36378/0 (socket says 172.21.15.119:36378) 2024-04-13T03:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5766d9d700 1 -- 172.21.15.119:0/4076460354 learned_addr learned my addr 172.21.15.119:0/4076460354 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5765d9b700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5768102c30 0x7f5768198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5766d9d700 1 -- 172.21.15.119:0/4076460354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5768102c30 msgr2=0x7f5768198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5766d9d700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5768102c30 0x7f5768198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5766d9d700 1 -- 172.21.15.119:0/4076460354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5768101a30 msgr2=0x7f5768197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5766d9d700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5768101a30 0x7f5768197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5766d9d700 1 -- 172.21.15.119:0/4076460354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576819f990 con 0x7f57681035d0 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5765d9b700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5768102c30 0x7f5768198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f5766d9d700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 0x7f576819f2c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f57600062a0 tx=0x7f5760007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.706+0000 7f575f7fe700 1 -- 172.21.15.119:0/4076460354 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760003c00 con 0x7f57681035d0 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.707+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576819fc20 con 0x7f57681035d0 2024-04-13T03:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.707+0000 7f575f7fe700 1 -- 172.21.15.119:0/4076460354 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5760003da0 con 0x7f57681035d0 2024-04-13T03:25:35.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.707+0000 7f575f7fe700 1 -- 172.21.15.119:0/4076460354 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57600055d0 con 0x7f57681035d0 2024-04-13T03:25:35.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.707+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57681a0100 con 0x7f57681035d0 2024-04-13T03:25:35.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.707+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57680697a0 con 0x7f57681035d0 2024-04-13T03:25:35.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.709+0000 7f575f7fe700 1 -- 172.21.15.119:0/4076460354 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f5760005770 con 0x7f57681035d0 2024-04-13T03:25:35.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.709+0000 7f575f7fe700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f57540a3bb0 0x7f57540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:35.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.709+0000 7f575f7fe700 1 -- 172.21.15.119:0/4076460354 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f576000f020 con 0x7f57681035d0 2024-04-13T03:25:35.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.710+0000 7f576659c700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f57540a3bb0 0x7f57540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:35.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.711+0000 7f576659c700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f57540a3bb0 0x7f57540a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5768199220 tx=0x7f5750006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:25:35.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.713+0000 7f575f7fe700 1 -- 172.21.15.119:0/4076460354 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f576002b020 con 0x7f57681035d0 2024-04-13T03:25:35.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:35.923+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5768062ef0 con 0x7f57540a3bb0 2024-04-13T03:25:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi016 2024-04-13T03:25:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:25:37.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi016 to 17094M 2024-04-13T03:25:37.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:37.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:37.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:25:37.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:25:37.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:25:37.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi016 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi016 to 17094M 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:25:37.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:25:37.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:25:37.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:37.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi016 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi016 to 17094M 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:25:37.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:25:37.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:25:37.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:39.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:38 smithi016 ceph-mon[31772]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:39.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:38 smithi016 ceph-mon[31772]: from='client.? 172.21.15.119:0/1744057402' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f603a5d2-28ba-439b-8855-6ff82db896aa"}]: dispatch 2024-04-13T03:25:39.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:38 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f603a5d2-28ba-439b-8855-6ff82db896aa"}]: dispatch 2024-04-13T03:25:39.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:38 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f603a5d2-28ba-439b-8855-6ff82db896aa"}]': finished 2024-04-13T03:25:39.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:38 smithi016 ceph-mon[31772]: osdmap e10: 2 total, 1 up, 2 in 2024-04-13T03:25:39.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:38 smithi119 ceph-mon[35835]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:38 smithi119 ceph-mon[35835]: from='client.? 172.21.15.119:0/1744057402' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f603a5d2-28ba-439b-8855-6ff82db896aa"}]: dispatch 2024-04-13T03:25:39.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:38 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f603a5d2-28ba-439b-8855-6ff82db896aa"}]: dispatch 2024-04-13T03:25:39.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:38 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f603a5d2-28ba-439b-8855-6ff82db896aa"}]': finished 2024-04-13T03:25:39.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:38 smithi119 ceph-mon[35835]: osdmap e10: 2 total, 1 up, 2 in 2024-04-13T03:25:39.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:39.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:38 smithi188 ceph-mon[36398]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:39.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:38 smithi188 ceph-mon[36398]: from='client.? 172.21.15.119:0/1744057402' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f603a5d2-28ba-439b-8855-6ff82db896aa"}]: dispatch 2024-04-13T03:25:39.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:38 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f603a5d2-28ba-439b-8855-6ff82db896aa"}]: dispatch 2024-04-13T03:25:39.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:38 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f603a5d2-28ba-439b-8855-6ff82db896aa"}]': finished 2024-04-13T03:25:39.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:38 smithi188 ceph-mon[36398]: osdmap e10: 2 total, 1 up, 2 in 2024-04-13T03:25:39.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:40.181 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:39 smithi016 ceph-mon[31772]: from='client.? 172.21.15.119:0/2768466919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:25:40.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:39 smithi119 ceph-mon[35835]: from='client.? 172.21.15.119:0/2768466919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:25:40.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:39 smithi188 ceph-mon[36398]: from='client.? 172.21.15.119:0/2768466919' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:25:41.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:40 smithi119 ceph-mon[35835]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:41.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:40 smithi016 ceph-mon[31772]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:41.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:40 smithi188 ceph-mon[36398]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:42 smithi119 ceph-mon[35835]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:43.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:42 smithi016 ceph-mon[31772]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:43.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:42 smithi188 ceph-mon[36398]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:45.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:44 smithi119 ceph-mon[35835]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:45.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:44 smithi016 ceph-mon[31772]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:45.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:44 smithi188 ceph-mon[36398]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:45 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-13T03:25:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:45 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:46.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:45 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-13T03:25:46.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:45 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:46.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:45 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-13T03:25:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:45 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:47.227 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:46 smithi119 ceph-mon[35835]: Deploying daemon osd.1 on smithi119 2024-04-13T03:25:47.227 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:46 smithi119 ceph-mon[35835]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:46 smithi016 ceph-mon[31772]: Deploying daemon osd.1 on smithi119 2024-04-13T03:25:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:46 smithi016 ceph-mon[31772]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:46 smithi188 ceph-mon[36398]: Deploying daemon osd.1 on smithi119 2024-04-13T03:25:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:46 smithi188 ceph-mon[36398]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:49.233 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:48 smithi119 ceph-mon[35835]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:48 smithi016 ceph-mon[31772]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:48 smithi188 ceph-mon[36398]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:50.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:50 smithi119 ceph-mon[35835]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:51.177 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:50 smithi188 ceph-mon[36398]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:51.244 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:50 smithi016 ceph-mon[31772]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:52 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:52 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:52 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:52.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:52 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:52.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:52 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:52.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:52 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:52.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:52 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:52.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:52 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:52.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:52 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:53.195 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:53 smithi119 ceph-mon[35835]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:53 smithi016 ceph-mon[31772]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:53 smithi188 ceph-mon[36398]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:54.602 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:54.602 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:54.602 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:54.602 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:54.602 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:54.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:54.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:54.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:54.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:54.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:54.673 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 1 on host 'smithi119' 2024-04-13T03:25:54.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.671+0000 7f575f7fe700 1 -- 172.21.15.119:0/4076460354 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5768062ef0 con 0x7f57540a3bb0 2024-04-13T03:25:54.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f57540a3bb0 msgr2=0x7f57540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f57540a3bb0 0x7f57540a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5768199220 tx=0x7f5750006d20 comp rx=0 tx=0).stop 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 msgr2=0x7f576819f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 0x7f576819f2c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f57600062a0 tx=0x7f5760007e40 comp rx=0 tx=0).stop 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 shutdown_connections 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f57540a3bb0 0x7f57540a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57681035d0 0x7f576819f2c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5768101a30 0x7f5768197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 --2- 172.21.15.119:0/4076460354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5768102c30 0x7f5768198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 >> 172.21.15.119:0/4076460354 conn(0x7f57680fd010 msgr2=0x7f5768105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 shutdown_connections 2024-04-13T03:25:54.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:25:54.674+0000 7f576cdd7700 1 -- 172.21.15.119:0/4076460354 wait complete. 2024-04-13T03:25:55.339 DEBUG:teuthology.orchestra.run.smithi119:osd.1> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.1.service 2024-04-13T03:25:55.341 INFO:tasks.cephadm:Deploying osd.2 on smithi188 with /dev/vg_nvme/lv_4... 2024-04-13T03:25:55.341 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-13T03:25:55.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:55 smithi119 ceph-mon[35835]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:55.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:55 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:55.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:55 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:55.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:55 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:55.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:55 smithi119 ceph-mon[35835]: from='osd.1 [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-13T03:25:55.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:55 smithi119 ceph-mon[35835]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-13T03:25:55.587 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:25:55.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:55 smithi188 ceph-mon[36398]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:55.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:55 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:55.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:55 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:55.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:55 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:55.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:55 smithi188 ceph-mon[36398]: from='osd.1 [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-13T03:25:55.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:55 smithi188 ceph-mon[36398]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-13T03:25:55.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:55 smithi016 ceph-mon[31772]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:55.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:55.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:55.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:55 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:55.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:55 smithi016 ceph-mon[31772]: from='osd.1 [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-13T03:25:55.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:55 smithi016 ceph-mon[31772]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-13T03:25:56.937 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:56 smithi188 ceph-mon[36398]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-13T03:25:56.938 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:56 smithi188 ceph-mon[36398]: from='osd.1 [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:25:56.938 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:56 smithi188 ceph-mon[36398]: osdmap e11: 2 total, 1 up, 2 in 2024-04-13T03:25:56.938 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:56 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:56.938 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:56 smithi188 ceph-mon[36398]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:25:56.938 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:56 smithi188 ceph-mon[36398]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:56.957 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:56 smithi119 ceph-mon[35835]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-13T03:25:56.958 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:56 smithi119 ceph-mon[35835]: from='osd.1 [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:25:56.958 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:56 smithi119 ceph-mon[35835]: osdmap e11: 2 total, 1 up, 2 in 2024-04-13T03:25:56.958 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:56 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:56.958 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:56 smithi119 ceph-mon[35835]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:25:56.958 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:56 smithi119 ceph-mon[35835]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:56.958 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:25:56 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1[40404]: 2024-04-13T03:25:56.658+0000 7fdf81b51700 -1 osd.1 0 waiting for initial osdmap 2024-04-13T03:25:56.958 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:25:56 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1[40404]: 2024-04-13T03:25:56.662+0000 7fdf7d944700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T03:25:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:56 smithi016 ceph-mon[31772]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-13T03:25:57.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:56 smithi016 ceph-mon[31772]: from='osd.1 [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:25:57.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:56 smithi016 ceph-mon[31772]: osdmap e11: 2 total, 1 up, 2 in 2024-04-13T03:25:57.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:56 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:57.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:56 smithi016 ceph-mon[31772]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:25:57.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:56 smithi016 ceph-mon[31772]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:57.379 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-13T03:25:57.380 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-13T03:25:57.380 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-13T03:25:57.380 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-13T03:25:57.380 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113399 s, 925 MB/s 2024-04-13T03:25:57.380 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-13T03:25:57.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:57 smithi119 ceph-mon[35835]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:25:57.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:57 smithi119 ceph-mon[35835]: osdmap e12: 2 total, 1 up, 2 in 2024-04-13T03:25:57.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:57 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:57.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:57 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:57.999 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:57 smithi188 ceph-mon[36398]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:25:57.999 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:57 smithi188 ceph-mon[36398]: osdmap e12: 2 total, 1 up, 2 in 2024-04-13T03:25:57.999 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:57 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:57.999 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:57 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:58.027 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2024-04-13T03:25:58.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:57 smithi016 ceph-mon[31772]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:25:58.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:57 smithi016 ceph-mon[31772]: osdmap e12: 2 total, 1 up, 2 in 2024-04-13T03:25:58.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:57 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:57 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:58.268 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:25:59.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:25:59.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:25:59.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:59.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: osd.1 [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] boot 2024-04-13T03:25:59.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: osdmap e13: 2 total, 2 up, 2 in 2024-04-13T03:25:59.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:59.063 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:59.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi119 2024-04-13T03:25:59.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:25:59.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi119 to 17080M 2024-04-13T03:25:59.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:59.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:59.064 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:25:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:25:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: osd.1 [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] boot 2024-04-13T03:25:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: osdmap e13: 2 total, 2 up, 2 in 2024-04-13T03:25:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi119 2024-04-13T03:25:59.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:25:59.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi119 to 17080M 2024-04-13T03:25:59.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:59.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:59.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: osd.1 [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] boot 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: osdmap e13: 2 total, 2 up, 2 in 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi119 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:25:59.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi119 to 17080M 2024-04-13T03:25:59.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:59.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:59.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:25:59.758 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:25:59 smithi188 ceph-mon[36398]: osdmap e14: 2 total, 2 up, 2 in 2024-04-13T03:25:59.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 -- 172.21.15.188:0/845540963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84105ed0 msgr2=0x7fee841082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:59.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 --2- 172.21.15.188:0/845540963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84105ed0 0x7fee841082c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fee70009990 tx=0x7fee70009ca0 comp rx=0 tx=0).stop 2024-04-13T03:25:59.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 -- 172.21.15.188:0/845540963 shutdown_connections 2024-04-13T03:25:59.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 --2- 172.21.15.188:0/845540963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee84108800 0x7fee8410ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:59.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 --2- 172.21.15.188:0/845540963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee841035a0 0x7fee84105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:59.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 --2- 172.21.15.188:0/845540963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84105ed0 0x7fee841082c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:59.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 -- 172.21.15.188:0/845540963 >> 172.21.15.188:0/845540963 conn(0x7fee840fcfb0 msgr2=0x7fee840ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:25:59.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 -- 172.21.15.188:0/845540963 shutdown_connections 2024-04-13T03:25:59.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 -- 172.21.15.188:0/845540963 wait complete. 2024-04-13T03:25:59.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 Processor -- start 2024-04-13T03:25:59.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.777+0000 7fee8af6c700 1 -- start start 2024-04-13T03:25:59.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee8af6c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee841035a0 0x7fee84198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:59.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee8af6c700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee84105ed0 0x7fee84199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:59.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee8af6c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84108800 0x7fee841a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:59.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee8af6c700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee8410e830 con 0x7fee84105ed0 2024-04-13T03:25:59.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee8af6c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fee8410e6b0 con 0x7fee841035a0 2024-04-13T03:25:59.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee8af6c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fee8410e9b0 con 0x7fee84108800 2024-04-13T03:25:59.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee89509700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84108800 0x7fee841a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:59.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee89509700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84108800 0x7fee841a04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:48462/0 (socket says 172.21.15.188:48462) 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee89509700 1 -- 172.21.15.188:0/2192813478 learned_addr learned my addr 172.21.15.188:0/2192813478 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee88d08700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee841035a0 0x7fee84198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee89509700 1 -- 172.21.15.188:0/2192813478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee841035a0 msgr2=0x7fee84198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee89509700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee841035a0 0x7fee84198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee89509700 1 -- 172.21.15.188:0/2192813478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee84105ed0 msgr2=0x7fee84199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee89509700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee84105ed0 0x7fee84199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee89509700 1 -- 172.21.15.188:0/2192813478 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee841a0b90 con 0x7fee84108800 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee88d08700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee841035a0 0x7fee84198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee89509700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84108800 0x7fee841a04f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00ba10 tx=0x7fee7c00bdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee81ffb700 1 -- 172.21.15.188:0/2192813478 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c019070 con 0x7fee84108800 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee81ffb700 1 -- 172.21.15.188:0/2192813478 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee7c014440 con 0x7fee84108800 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee81ffb700 1 -- 172.21.15.188:0/2192813478 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c0125a0 con 0x7fee84108800 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.778+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee841a0e80 con 0x7fee84108800 2024-04-13T03:25:59.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.779+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee841a1360 con 0x7fee84108800 2024-04-13T03:25:59.790 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.779+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee840697e0 con 0x7fee84108800 2024-04-13T03:25:59.790 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.780+0000 7fee81ffb700 1 -- 172.21.15.188:0/2192813478 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fee7c01c070 con 0x7fee84108800 2024-04-13T03:25:59.790 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.781+0000 7fee81ffb700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee740a3c30 0x7fee740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:25:59.790 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.781+0000 7fee81ffb700 1 -- 172.21.15.188:0/2192813478 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fee7c0cde20 con 0x7fee84108800 2024-04-13T03:25:59.790 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.782+0000 7fee88d08700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee740a3c30 0x7fee740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:25:59.790 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.783+0000 7fee88d08700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee740a3c30 0x7fee740a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee78007900 tx=0x7fee78008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:25:59.790 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.786+0000 7fee81ffb700 1 -- 172.21.15.188:0/2192813478 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee7c09c0e0 con 0x7fee84108800 2024-04-13T03:25:59.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:25:59.989+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fee84062ef0 con 0x7fee740a3c30 2024-04-13T03:26:00.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:25:59 smithi119 ceph-mon[35835]: osdmap e14: 2 total, 2 up, 2 in 2024-04-13T03:26:00.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:25:59 smithi016 ceph-mon[31772]: osdmap e14: 2 total, 2 up, 2 in 2024-04-13T03:26:00.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:00 smithi188 ceph-mon[36398]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:00.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:00 smithi188 ceph-mon[36398]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:26:00.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:26:00.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:26:00.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:00.972 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:00 smithi119 ceph-mon[35835]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:00.972 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:00 smithi119 ceph-mon[35835]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:26:00.973 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:26:00.973 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:26:00.973 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:00 smithi016 ceph-mon[31772]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:00 smithi016 ceph-mon[31772]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:26:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:26:01.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:26:01.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:02 smithi119 ceph-mon[35835]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:02 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/2271157541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c288439-37d1-43e9-8e52-9a08bdbcfff9"}]: dispatch 2024-04-13T03:26:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:02 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c288439-37d1-43e9-8e52-9a08bdbcfff9"}]: dispatch 2024-04-13T03:26:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:02 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c288439-37d1-43e9-8e52-9a08bdbcfff9"}]': finished 2024-04-13T03:26:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:02 smithi119 ceph-mon[35835]: osdmap e15: 3 total, 2 up, 3 in 2024-04-13T03:26:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:02 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:02 smithi016 ceph-mon[31772]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:02 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/2271157541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c288439-37d1-43e9-8e52-9a08bdbcfff9"}]: dispatch 2024-04-13T03:26:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:02 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c288439-37d1-43e9-8e52-9a08bdbcfff9"}]: dispatch 2024-04-13T03:26:03.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:02 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c288439-37d1-43e9-8e52-9a08bdbcfff9"}]': finished 2024-04-13T03:26:03.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:02 smithi016 ceph-mon[31772]: osdmap e15: 3 total, 2 up, 3 in 2024-04-13T03:26:03.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:02 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:02 smithi188 ceph-mon[36398]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:02 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/2271157541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c288439-37d1-43e9-8e52-9a08bdbcfff9"}]: dispatch 2024-04-13T03:26:03.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:02 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c288439-37d1-43e9-8e52-9a08bdbcfff9"}]: dispatch 2024-04-13T03:26:03.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:02 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c288439-37d1-43e9-8e52-9a08bdbcfff9"}]': finished 2024-04-13T03:26:03.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:02 smithi188 ceph-mon[36398]: osdmap e15: 3 total, 2 up, 3 in 2024-04-13T03:26:03.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:02 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:04.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:03 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/2982989965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:26:04.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:03 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/2982989965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:26:04.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:03 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/2982989965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:26:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:04 smithi016 ceph-mon[31772]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:04 smithi188 ceph-mon[36398]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:04 smithi119 ceph-mon[35835]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:07.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:06 smithi016 ceph-mon[31772]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:06 smithi188 ceph-mon[36398]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:06 smithi119 ceph-mon[35835]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:08.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:08 smithi188 ceph-mon[36398]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:08 smithi016 ceph-mon[31772]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:08 smithi119 ceph-mon[35835]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:10.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:09 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-13T03:26:10.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:09 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:10.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:09 smithi016 ceph-mon[31772]: Deploying daemon osd.2 on smithi188 2024-04-13T03:26:10.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:09 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-13T03:26:10.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:09 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:10.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:09 smithi188 ceph-mon[36398]: Deploying daemon osd.2 on smithi188 2024-04-13T03:26:10.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:09 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-13T03:26:10.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:09 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:10.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:09 smithi119 ceph-mon[35835]: Deploying daemon osd.2 on smithi188 2024-04-13T03:26:10.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:10 smithi188 ceph-mon[36398]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:10 smithi016 ceph-mon[31772]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:10 smithi119 ceph-mon[35835]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:12 smithi119 ceph-mon[35835]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:13.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:12 smithi188 ceph-mon[36398]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:12 smithi016 ceph-mon[31772]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:15.560 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:15 smithi188 ceph-mon[36398]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:15.560 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:15 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:15.561 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:15 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:15.561 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:15 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:15 smithi119 ceph-mon[35835]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:15 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:15 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:15 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:15.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:15 smithi016 ceph-mon[31772]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:15.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:15 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:15.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:15 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:15.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:15 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:17 smithi188 ceph-mon[36398]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:17 smithi119 ceph-mon[35835]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:17 smithi016 ceph-mon[31772]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:17.722 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.721+0000 7fee81ffb700 1 -- 172.21.15.188:0/2192813478 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fee84062ef0 con 0x7fee740a3c30 2024-04-13T03:26:17.722 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 2 on host 'smithi188' 2024-04-13T03:26:17.724 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee740a3c30 msgr2=0x7fee740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee740a3c30 0x7fee740a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee78007900 tx=0x7fee78008040 comp rx=0 tx=0).stop 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84108800 msgr2=0x7fee841a04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84108800 0x7fee841a04f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00ba10 tx=0x7fee7c00bdd0 comp rx=0 tx=0).stop 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 shutdown_connections 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee740a3c30 0x7fee740a60f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee84108800 0x7fee841a04f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee841035a0 0x7fee84198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 --2- 172.21.15.188:0/2192813478 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee84105ed0 0x7fee84199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 >> 172.21.15.188:0/2192813478 conn(0x7fee840fcfb0 msgr2=0x7fee84104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 shutdown_connections 2024-04-13T03:26:17.725 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:26:17.724+0000 7fee8af6c700 1 -- 172.21.15.188:0/2192813478 wait complete. 2024-04-13T03:26:18.324 DEBUG:teuthology.orchestra.run.smithi188:osd.2> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.2.service 2024-04-13T03:26:18.327 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_3... 2024-04-13T03:26:18.327 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-13T03:26:18.790 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:26:18.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:18.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:18.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:18.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:18 smithi188 ceph-mon[36398]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:18.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:18 smithi188 ceph-mon[36398]: from='osd.2 [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-13T03:26:18.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:18 smithi188 ceph-mon[36398]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-13T03:26:18.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:18.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:18.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:18.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:18 smithi016 ceph-mon[31772]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:18.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:18 smithi016 ceph-mon[31772]: from='osd.2 [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-13T03:26:18.936 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:18 smithi016 ceph-mon[31772]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-13T03:26:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:18 smithi119 ceph-mon[35835]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:18 smithi119 ceph-mon[35835]: from='osd.2 [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-13T03:26:19.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:18 smithi119 ceph-mon[35835]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-13T03:26:20.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:19 smithi016 ceph-mon[31772]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-13T03:26:20.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:19 smithi016 ceph-mon[31772]: osdmap e16: 3 total, 2 up, 3 in 2024-04-13T03:26:20.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:19 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:20.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:19 smithi016 ceph-mon[31772]: from='osd.2 [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:26:20.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:19 smithi016 ceph-mon[31772]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:26:20.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:19 smithi119 ceph-mon[35835]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-13T03:26:20.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:19 smithi119 ceph-mon[35835]: osdmap e16: 3 total, 2 up, 3 in 2024-04-13T03:26:20.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:19 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:20.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:19 smithi119 ceph-mon[35835]: from='osd.2 [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:26:20.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:19 smithi119 ceph-mon[35835]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:26:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:19 smithi188 ceph-mon[36398]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-13T03:26:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:19 smithi188 ceph-mon[36398]: osdmap e16: 3 total, 2 up, 3 in 2024-04-13T03:26:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:19 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:19 smithi188 ceph-mon[36398]: from='osd.2 [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:26:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:19 smithi188 ceph-mon[36398]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:26:20.534 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:20 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2[41520]: 2024-04-13T03:26:20.163+0000 7f6a5007c700 -1 osd.2 0 waiting for initial osdmap 2024-04-13T03:26:20.534 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:20 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2[41520]: 2024-04-13T03:26:20.171+0000 7f6a47e67700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T03:26:20.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:20 smithi188 ceph-mon[36398]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:26:20.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:20 smithi188 ceph-mon[36398]: osdmap e17: 3 total, 2 up, 3 in 2024-04-13T03:26:20.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:20.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:20.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:20 smithi188 ceph-mon[36398]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:20.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:20 smithi188 ceph-mon[36398]: from='osd.2 ' entity='osd.2' 2024-04-13T03:26:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:20 smithi119 ceph-mon[35835]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:26:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:20 smithi119 ceph-mon[35835]: osdmap e17: 3 total, 2 up, 3 in 2024-04-13T03:26:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:20 smithi119 ceph-mon[35835]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:21.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:20 smithi119 ceph-mon[35835]: from='osd.2 ' entity='osd.2' 2024-04-13T03:26:21.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:20 smithi016 ceph-mon[31772]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:26:21.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:20 smithi016 ceph-mon[31772]: osdmap e17: 3 total, 2 up, 3 in 2024-04-13T03:26:21.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:21.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:21.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:20 smithi016 ceph-mon[31772]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-13T03:26:21.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:20 smithi016 ceph-mon[31772]: from='osd.2 ' entity='osd.2' 2024-04-13T03:26:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-13T03:26:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-13T03:26:21.532 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-13T03:26:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-13T03:26:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0324325 s, 323 MB/s 2024-04-13T03:26:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-13T03:26:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:26:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:26:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi188 2024-04-13T03:26:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi188 to 17080M 2024-04-13T03:26:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: osd.2 [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] boot 2024-04-13T03:26:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: osdmap e18: 3 total, 3 up, 3 in 2024-04-13T03:26:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:22.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:26:22.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:26:22.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi188 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi188 to 17080M 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: osd.2 [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] boot 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: osdmap e18: 3 total, 3 up, 3 in 2024-04-13T03:26:22.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi188 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi188 to 17080M 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:22.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:22.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:22.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:22.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: osd.2 [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] boot 2024-04-13T03:26:22.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: osdmap e18: 3 total, 3 up, 3 in 2024-04-13T03:26:22.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:26:22.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-04-13T03:26:22.881 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:26:23.029 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:22 smithi016 ceph-mon[31772]: pgmap v89: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:23.029 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:22 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:26:23.029 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:22 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-13T03:26:23.029 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:22 smithi016 ceph-mon[31772]: osdmap e19: 3 total, 3 up, 3 in 2024-04-13T03:26:23.029 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:22 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:26:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:22 smithi119 ceph-mon[35835]: pgmap v89: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:22 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:26:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:22 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-13T03:26:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:22 smithi119 ceph-mon[35835]: osdmap e19: 3 total, 3 up, 3 in 2024-04-13T03:26:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:22 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:26:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:22 smithi188 ceph-mon[36398]: pgmap v89: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:23.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:22 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:26:23.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:22 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-13T03:26:23.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:22 smithi188 ceph-mon[36398]: osdmap e19: 3 total, 3 up, 3 in 2024-04-13T03:26:23.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:22 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:26:23.624 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48086]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T03:26:23.624 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48086]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:23.624 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48086]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:23.624 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48086]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:23.625 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48089]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T03:26:23.625 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48089]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:23.625 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48089]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:23.625 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48089]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:23.872 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45313]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T03:26:23.873 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45313]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:23.873 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45313]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:23.873 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45313]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:23.873 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45316]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T03:26:23.873 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45316]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:23.873 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45316]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:23.873 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45316]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:24.002 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:26:23 smithi119 sudo[44109]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T03:26:24.002 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:26:23 smithi119 sudo[44109]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:24.002 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:26:23 smithi119 sudo[44109]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:24.002 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:26:23 smithi119 sudo[44109]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:24.002 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:26:23 smithi119 sudo[44112]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T03:26:24.002 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:26:23 smithi119 sudo[44112]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:24.002 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:26:23 smithi119 sudo[44112]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:24.002 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:26:23 smithi119 sudo[44112]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:24.199 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48095]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T03:26:24.199 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48095]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:24.199 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48095]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:24.199 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48095]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:24.199 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48098]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T03:26:24.199 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48098]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:24.199 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48098]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:24.199 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:23 smithi016 sudo[48098]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:24.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 sudo[44115]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T03:26:24.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 sudo[44115]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 sudo[44115]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 sudo[44115]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 sudo[44118]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 sudo[44118]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 sudo[44118]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 sudo[44118]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: osdmap e20: 3 total, 3 up, 3 in 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:24.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:26:24.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:26:24.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:26:24.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:24 smithi119 ceph-mon[35835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45319]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45319]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45319]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45319]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45322]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45322]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45322]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:23 smithi188 sudo[45322]: pam_unix(sudo:session): session closed for user root 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: osdmap e20: 3 total, 3 up, 3 in 2024-04-13T03:26:24.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T03:26:24.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T03:26:24.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:26:24.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:26:24.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:26:24.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:24 smithi188 ceph-mon[36398]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: osdmap e20: 3 total, 3 up, 3 in 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:26:24.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T03:26:24.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:26:24.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:26:24.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:26:24.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T03:26:24.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T03:26:24.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:26:24.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:26:24.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:26:24.590 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:24 smithi016 ceph-mon[31772]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T03:26:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.119+0000 7fe08a200700 1 -- 172.21.15.16:0/2221877308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe084101cf0 msgr2=0x7fe084102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.119+0000 7fe08a200700 1 --2- 172.21.15.16:0/2221877308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe084101cf0 0x7fe084102170 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe070009990 tx=0x7fe070009ca0 comp rx=0 tx=0).stop 2024-04-13T03:26:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.119+0000 7fe08a200700 1 -- 172.21.15.16:0/2221877308 shutdown_connections 2024-04-13T03:26:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.119+0000 7fe08a200700 1 --2- 172.21.15.16:0/2221877308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0841026b0 0x7fe084102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.119+0000 7fe08a200700 1 --2- 172.21.15.16:0/2221877308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101390 0x7fe0841017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:25.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.119+0000 7fe08a200700 1 --2- 172.21.15.16:0/2221877308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe084101cf0 0x7fe084102170 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:25.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.119+0000 7fe08a200700 1 -- 172.21.15.16:0/2221877308 >> 172.21.15.16:0/2221877308 conn(0x7fe0840fcfb0 msgr2=0x7fe0840ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:26:25.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.120+0000 7fe08a200700 1 -- 172.21.15.16:0/2221877308 shutdown_connections 2024-04-13T03:26:25.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.120+0000 7fe08a200700 1 -- 172.21.15.16:0/2221877308 wait complete. 2024-04-13T03:26:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe08a200700 1 Processor -- start 2024-04-13T03:26:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe08a200700 1 -- start start 2024-04-13T03:26:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe08a200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101390 0x7fe08419af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:26:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe08a200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe084101cf0 0x7fe08419b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:26:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe08a200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0841026b0 0x7fe0841a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:26:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe08a200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841108c0 con 0x7fe084101390 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe08a200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe084110740 con 0x7fe0841026b0 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe08a200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe084110a40 con 0x7fe084101cf0 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe083400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101390 0x7fe08419af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe083400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101390 0x7fe08419af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43494/0 (socket says 172.21.15.16:43494) 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe083400700 1 -- 172.21.15.16:0/1403693513 learned_addr learned my addr 172.21.15.16:0/1403693513 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.121+0000 7fe082a00700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe084101cf0 0x7fe08419b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.122+0000 7fe083e00700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0841026b0 0x7fe0841a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.122+0000 7fe083400700 1 -- 172.21.15.16:0/1403693513 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe084101cf0 msgr2=0x7fe08419b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.122+0000 7fe083400700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe084101cf0 0x7fe08419b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.122+0000 7fe083400700 1 -- 172.21.15.16:0/1403693513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0841026b0 msgr2=0x7fe0841a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.122+0000 7fe083400700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0841026b0 0x7fe0841a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.122+0000 7fe083400700 1 -- 172.21.15.16:0/1403693513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0841a2be0 con 0x7fe084101390 2024-04-13T03:26:25.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.122+0000 7fe083400700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101390 0x7fe08419af80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe07400e9d0 tx=0x7fe07400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:26:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.123+0000 7fe07be00700 1 -- 172.21.15.16:0/1403693513 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07400cbd0 con 0x7fe084101390 2024-04-13T03:26:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.123+0000 7fe07be00700 1 -- 172.21.15.16:0/1403693513 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe07400ee70 con 0x7fe084101390 2024-04-13T03:26:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.123+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0841a2ed0 con 0x7fe084101390 2024-04-13T03:26:25.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.123+0000 7fe07be00700 1 -- 172.21.15.16:0/1403693513 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074018680 con 0x7fe084101390 2024-04-13T03:26:25.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.123+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0841a3330 con 0x7fe084101390 2024-04-13T03:26:25.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.124+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0840697a0 con 0x7fe084101390 2024-04-13T03:26:25.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.126+0000 7fe07be00700 1 -- 172.21.15.16:0/1403693513 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe0740188e0 con 0x7fe084101390 2024-04-13T03:26:25.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.128+0000 7fe07be00700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe0680a3cc0 0x7fe0680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:26:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.128+0000 7fe07be00700 1 -- 172.21.15.16:0/1403693513 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7fe0740cf240 con 0x7fe084101390 2024-04-13T03:26:25.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.128+0000 7fe082a00700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe0680a3cc0 0x7fe0680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:26:25.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.129+0000 7fe082a00700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe0680a3cc0 0x7fe0680a6180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe07000b7b0 tx=0x7fe07000b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:26:25.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.133+0000 7fe07be00700 1 -- 172.21.15.16:0/1403693513 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe07409cf50 con 0x7fe084101390 2024-04-13T03:26:25.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:25.379+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe084062ef0 con 0x7fe0680a3cc0 2024-04-13T03:26:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:25 smithi119 ceph-mon[35835]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:25 smithi119 ceph-mon[35835]: osdmap e21: 3 total, 3 up, 3 in 2024-04-13T03:26:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:25 smithi119 ceph-mon[35835]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-13T03:26:25.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:25 smithi016 ceph-mon[31772]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:25.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:25 smithi016 ceph-mon[31772]: osdmap e21: 3 total, 3 up, 3 in 2024-04-13T03:26:25.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:25 smithi016 ceph-mon[31772]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-13T03:26:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:25 smithi188 ceph-mon[36398]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:25 smithi188 ceph-mon[36398]: osdmap e21: 3 total, 3 up, 3 in 2024-04-13T03:26:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:25 smithi188 ceph-mon[36398]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-13T03:26:26.521 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:26 smithi016 ceph-mon[31772]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:26:26.521 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:26:26.522 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:26:26.522 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:26.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:26 smithi119 ceph-mon[35835]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:26:26.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:26:26.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:26:26.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:26.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:26 smithi188 ceph-mon[36398]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:26:26.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:26:26.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:26:26.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:27 smithi119 ceph-mon[35835]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:27.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:27 smithi016 ceph-mon[31772]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:27.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:27 smithi188 ceph-mon[36398]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:29.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:29 smithi119 ceph-mon[35835]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:29.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:29 smithi016 ceph-mon[31772]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:29.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:29 smithi188 ceph-mon[36398]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:30.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:30 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3538702639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a289feb8-b06b-4faf-b5f1-bf88a3cc8b88"}]: dispatch 2024-04-13T03:26:30.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:30 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a289feb8-b06b-4faf-b5f1-bf88a3cc8b88"}]: dispatch 2024-04-13T03:26:30.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:30 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a289feb8-b06b-4faf-b5f1-bf88a3cc8b88"}]': finished 2024-04-13T03:26:30.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:30 smithi016 ceph-mon[31772]: osdmap e22: 4 total, 3 up, 4 in 2024-04-13T03:26:30.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:30.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:30 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3538702639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a289feb8-b06b-4faf-b5f1-bf88a3cc8b88"}]: dispatch 2024-04-13T03:26:30.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:30 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a289feb8-b06b-4faf-b5f1-bf88a3cc8b88"}]: dispatch 2024-04-13T03:26:30.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:30 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a289feb8-b06b-4faf-b5f1-bf88a3cc8b88"}]': finished 2024-04-13T03:26:30.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:30 smithi119 ceph-mon[35835]: osdmap e22: 4 total, 3 up, 4 in 2024-04-13T03:26:30.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:30.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:30 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3538702639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a289feb8-b06b-4faf-b5f1-bf88a3cc8b88"}]: dispatch 2024-04-13T03:26:30.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:30 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a289feb8-b06b-4faf-b5f1-bf88a3cc8b88"}]: dispatch 2024-04-13T03:26:30.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:30 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a289feb8-b06b-4faf-b5f1-bf88a3cc8b88"}]': finished 2024-04-13T03:26:30.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:30 smithi188 ceph-mon[36398]: osdmap e22: 4 total, 3 up, 4 in 2024-04-13T03:26:30.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:31.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:31 smithi016 ceph-mon[31772]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2024-04-13T03:26:31.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:31 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/546282490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:26:31.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:31 smithi119 ceph-mon[35835]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2024-04-13T03:26:31.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:31 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/546282490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:26:31.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:31 smithi188 ceph-mon[36398]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2024-04-13T03:26:31.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:31 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/546282490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:26:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:33 smithi188 ceph-mon[36398]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-04-13T03:26:33.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:33 smithi119 ceph-mon[35835]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-04-13T03:26:33.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:33 smithi016 ceph-mon[31772]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-04-13T03:26:35.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:35 smithi016 ceph-mon[31772]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 136 KiB/s wr, 9 op/s 2024-04-13T03:26:35.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:35 smithi119 ceph-mon[35835]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 136 KiB/s wr, 9 op/s 2024-04-13T03:26:35.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:35 smithi188 ceph-mon[36398]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 136 KiB/s wr, 9 op/s 2024-04-13T03:26:37.336 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:37 smithi016 ceph-mon[31772]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-13T03:26:37.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:37 smithi119 ceph-mon[35835]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-13T03:26:37.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:37 smithi188 ceph-mon[36398]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-13T03:26:39.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:39 smithi119 ceph-mon[35835]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-13T03:26:39.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:39 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-13T03:26:39.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:39 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:39 smithi016 ceph-mon[31772]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-13T03:26:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-13T03:26:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:39.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:39 smithi188 ceph-mon[36398]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-13T03:26:39.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:39 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-13T03:26:39.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:39 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:40.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:40 smithi016 ceph-mon[31772]: Deploying daemon osd.3 on smithi016 2024-04-13T03:26:40.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:40 smithi119 ceph-mon[35835]: Deploying daemon osd.3 on smithi016 2024-04-13T03:26:40.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:40 smithi188 ceph-mon[36398]: Deploying daemon osd.3 on smithi016 2024-04-13T03:26:41.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:41 smithi119 ceph-mon[35835]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:41.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:41 smithi016 ceph-mon[31772]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:41.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:41 smithi188 ceph-mon[36398]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:43.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:43 smithi119 ceph-mon[35835]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:43 smithi016 ceph-mon[31772]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:43.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:43 smithi188 ceph-mon[36398]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:45.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:45 smithi119 ceph-mon[35835]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:45.591 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:45 smithi016 ceph-mon[31772]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:45.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:45 smithi188 ceph-mon[36398]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:47 smithi016 ceph-mon[31772]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:47.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:47 smithi119 ceph-mon[35835]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:47.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:47 smithi188 ceph-mon[36398]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:48.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:48.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:48.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:48.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:48.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:48.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:48.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:48 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:48.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:48 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:48.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:48 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:49.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:49 smithi016 ceph-mon[31772]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:49.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:49 smithi188 ceph-mon[36398]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:49.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:49 smithi119 ceph-mon[35835]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:51.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:51 smithi016 ceph-mon[31772]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:51.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:51.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:51.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:51.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:51.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:51.566 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:51 smithi016 ceph-mon[31772]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-13T03:26:51.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:51 smithi188 ceph-mon[36398]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:51.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:51.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:51.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:51.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:51.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:51.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:51 smithi188 ceph-mon[36398]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-13T03:26:51.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:51 smithi119 ceph-mon[35835]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:51.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:51.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:51.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:51.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:51.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:51.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:51 smithi119 ceph-mon[35835]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-13T03:26:52.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.477+0000 7fe07be00700 1 -- 172.21.15.16:0/1403693513 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe084062ef0 con 0x7fe0680a3cc0 2024-04-13T03:26:52.479 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-04-13T03:26:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe0680a3cc0 msgr2=0x7fe0680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe0680a3cc0 0x7fe0680a6180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe07000b7b0 tx=0x7fe07000b880 comp rx=0 tx=0).stop 2024-04-13T03:26:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101390 msgr2=0x7fe08419af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101390 0x7fe08419af80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe07400e9d0 tx=0x7fe07400ece0 comp rx=0 tx=0).stop 2024-04-13T03:26:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 shutdown_connections 2024-04-13T03:26:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe0680a3cc0 0x7fe0680a6180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0841026b0 0x7fe0841a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe084101390 0x7fe08419af80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 --2- 172.21.15.16:0/1403693513 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe084101cf0 0x7fe08419b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.481+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 >> 172.21.15.16:0/1403693513 conn(0x7fe0840fcfb0 msgr2=0x7fe0840feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:26:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.482+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 shutdown_connections 2024-04-13T03:26:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:26:52.482+0000 7fe08a200700 1 -- 172.21.15.16:0/1403693513 wait complete. 2024-04-13T03:26:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:52 smithi119 ceph-mon[35835]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:52 smithi119 ceph-mon[35835]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-13T03:26:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:52 smithi119 ceph-mon[35835]: osdmap e23: 4 total, 3 up, 4 in 2024-04-13T03:26:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:52 smithi119 ceph-mon[35835]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:26:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:52 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:53.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:52 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:53.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:52 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:53.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:52 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:52 smithi016 ceph-mon[31772]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:52 smithi016 ceph-mon[31772]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-13T03:26:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:52 smithi016 ceph-mon[31772]: osdmap e23: 4 total, 3 up, 4 in 2024-04-13T03:26:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:52 smithi016 ceph-mon[31772]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:26:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:52 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:53.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:52 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:53.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:52 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:53.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:52 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:52 smithi188 ceph-mon[36398]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:52 smithi188 ceph-mon[36398]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-13T03:26:53.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:52 smithi188 ceph-mon[36398]: osdmap e23: 4 total, 3 up, 4 in 2024-04-13T03:26:53.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:52 smithi188 ceph-mon[36398]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:26:53.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:52 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:53.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:52 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:53.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:52 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:26:53.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:52 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:53.280 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.3.service 2024-04-13T03:26:53.286 INFO:tasks.cephadm:Deploying osd.4 on smithi119 with /dev/vg_nvme/lv_3... 2024-04-13T03:26:53.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-13T03:26:53.531 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:26:53.969 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:53 smithi119 ceph-mon[35835]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:26:53.970 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:53 smithi119 ceph-mon[35835]: osdmap e24: 4 total, 3 up, 4 in 2024-04-13T03:26:53.970 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:53 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:53.970 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:53 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:54.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:53 smithi016 ceph-mon[31772]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:26:54.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:53 smithi016 ceph-mon[31772]: osdmap e24: 4 total, 3 up, 4 in 2024-04-13T03:26:54.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:53 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:54.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:53 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:54.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:53 smithi188 ceph-mon[36398]: from='osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:26:54.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:53 smithi188 ceph-mon[36398]: osdmap e24: 4 total, 3 up, 4 in 2024-04-13T03:26:54.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:53 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:54.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:53 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:54.937 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:54 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:26:54.938 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:54 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:26:54.938 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:54 smithi119 ceph-mon[35835]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:54.938 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:54.938 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:54 smithi119 ceph-mon[35835]: osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] boot 2024-04-13T03:26:54.938 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:54 smithi119 ceph-mon[35835]: osdmap e25: 4 total, 4 up, 4 in 2024-04-13T03:26:54.938 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:55.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:54 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:26:55.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:54 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:26:55.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:54 smithi016 ceph-mon[31772]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:55.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:55.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:54 smithi016 ceph-mon[31772]: osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] boot 2024-04-13T03:26:55.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:54 smithi016 ceph-mon[31772]: osdmap e25: 4 total, 4 up, 4 in 2024-04-13T03:26:55.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:54 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:26:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:54 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:26:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:54 smithi188 ceph-mon[36398]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-13T03:26:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:54 smithi188 ceph-mon[36398]: osd.3 [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] boot 2024-04-13T03:26:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:54 smithi188 ceph-mon[36398]: osdmap e25: 4 total, 4 up, 4 in 2024-04-13T03:26:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:26:55.501 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-13T03:26:55.501 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-13T03:26:55.501 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-04-13T03:26:55.501 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-04-13T03:26:55.501 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0167389 s, 626 MB/s 2024-04-13T03:26:55.501 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-04-13T03:26:56.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:55 smithi016 ceph-mon[31772]: osdmap e26: 4 total, 4 up, 4 in 2024-04-13T03:26:56.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:55 smithi188 ceph-mon[36398]: osdmap e26: 4 total, 4 up, 4 in 2024-04-13T03:26:56.232 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:55 smithi119 ceph-mon[35835]: osdmap e26: 4 total, 4 up, 4 in 2024-04-13T03:26:56.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_3 2024-04-13T03:26:56.499 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:26:57.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:56 smithi016 ceph-mon[31772]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:26:57.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:56 smithi016 ceph-mon[31772]: osdmap e27: 4 total, 4 up, 4 in 2024-04-13T03:26:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:56 smithi188 ceph-mon[36398]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:26:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:56 smithi188 ceph-mon[36398]: osdmap e27: 4 total, 4 up, 4 in 2024-04-13T03:26:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:56 smithi119 ceph-mon[35835]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:26:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:56 smithi119 ceph-mon[35835]: osdmap e27: 4 total, 4 up, 4 in 2024-04-13T03:26:58.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:57 smithi188 ceph-mon[36398]: osdmap e28: 4 total, 4 up, 4 in 2024-04-13T03:26:58.125 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:57 smithi119 ceph-mon[35835]: osdmap e28: 4 total, 4 up, 4 in 2024-04-13T03:26:58.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.127+0000 7f1a983c6700 1 -- 172.21.15.119:0/274575650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90104740 msgr2=0x7f1a9010aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:58.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.127+0000 7f1a983c6700 1 --2- 172.21.15.119:0/274575650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90104740 0x7f1a9010aff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800b0a0 tx=0x7f1a8800b3b0 comp rx=0 tx=0).stop 2024-04-13T03:26:58.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.127+0000 7f1a983c6700 1 -- 172.21.15.119:0/274575650 shutdown_connections 2024-04-13T03:26:58.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.127+0000 7f1a983c6700 1 --2- 172.21.15.119:0/274575650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90104740 0x7f1a9010aff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:58.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.127+0000 7f1a983c6700 1 --2- 172.21.15.119:0/274575650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a90100dc0 0x7f1a901011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:58.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.127+0000 7f1a983c6700 1 --2- 172.21.15.119:0/274575650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a90103da0 0x7f1a90104200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:58.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.127+0000 7f1a983c6700 1 -- 172.21.15.119:0/274575650 >> 172.21.15.119:0/274575650 conn(0x7f1a900fc740 msgr2=0x7f1a900feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:26:58.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.127+0000 7f1a983c6700 1 -- 172.21.15.119:0/274575650 shutdown_connections 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.127+0000 7f1a983c6700 1 -- 172.21.15.119:0/274575650 wait complete. 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a983c6700 1 Processor -- start 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a983c6700 1 -- start start 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a983c6700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a90100dc0 0x7f1a90198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a983c6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90103da0 0x7f1a901992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a983c6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a90104740 0x7f1a901a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a983c6700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9010e7d0 con 0x7f1a90100dc0 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a983c6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9010e650 con 0x7f1a90103da0 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a983c6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9010e950 con 0x7f1a90104740 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a95961700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90103da0 0x7f1a901992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a95961700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90103da0 0x7f1a901992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60098/0 (socket says 172.21.15.119:60098) 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a95961700 1 -- 172.21.15.119:0/1947093713 learned_addr learned my addr 172.21.15.119:0/1947093713 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a96963700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a90104740 0x7f1a901a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a96162700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a90100dc0 0x7f1a90198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:26:58.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a95961700 1 -- 172.21.15.119:0/1947093713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a90104740 msgr2=0x7f1a901a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a95961700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a90104740 0x7f1a901a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a95961700 1 -- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a90100dc0 msgr2=0x7f1a90198d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a95961700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a90100dc0 0x7f1a90198d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a95961700 1 -- 172.21.15.119:0/1947093713 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a901a09c0 con 0x7f1a90103da0 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.128+0000 7f1a95961700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90103da0 0x7f1a901992a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000b530 tx=0x7f1a8000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.129+0000 7f1a877fe700 1 -- 172.21.15.119:0/1947093713 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80015020 con 0x7f1a90103da0 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.129+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a901a0cb0 con 0x7f1a90103da0 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.129+0000 7f1a877fe700 1 -- 172.21.15.119:0/1947093713 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a80011d10 con 0x7f1a90103da0 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.129+0000 7f1a877fe700 1 -- 172.21.15.119:0/1947093713 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000f2e0 con 0x7f1a90103da0 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.129+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a901a1260 con 0x7f1a90103da0 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.130+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a900697a0 con 0x7f1a90103da0 2024-04-13T03:26:58.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.131+0000 7f1a877fe700 1 -- 172.21.15.119:0/1947093713 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1a80018020 con 0x7f1a90103da0 2024-04-13T03:26:58.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.131+0000 7f1a877fe700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1a7c0a3c40 0x7f1a7c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:26:58.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.131+0000 7f1a877fe700 1 -- 172.21.15.119:0/1947093713 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7f1a800ce9d0 con 0x7f1a90103da0 2024-04-13T03:26:58.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.132+0000 7f1a96162700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1a7c0a3c40 0x7f1a7c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:26:58.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.132+0000 7f1a96162700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1a7c0a3c40 0x7f1a7c0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c006fd0 tx=0x7f1a8c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:26:58.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.136+0000 7f1a877fe700 1 -- 172.21.15.119:0/1947093713 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a8001f020 con 0x7f1a90103da0 2024-04-13T03:26:58.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:26:58.339+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1a90062ef0 con 0x7f1a7c0a3c40 2024-04-13T03:26:58.343 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:57 smithi016 ceph-mon[31772]: osdmap e28: 4 total, 4 up, 4 in 2024-04-13T03:26:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:58 smithi016 ceph-mon[31772]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:26:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:58 smithi016 ceph-mon[31772]: osdmap e29: 4 total, 4 up, 4 in 2024-04-13T03:26:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:58 smithi016 ceph-mon[31772]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:26:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:26:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:26:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:26:58 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:59.178 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:58 smithi119 ceph-mon[35835]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:26:59.179 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:58 smithi119 ceph-mon[35835]: osdmap e29: 4 total, 4 up, 4 in 2024-04-13T03:26:59.179 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:58 smithi119 ceph-mon[35835]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:26:59.179 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:26:59.179 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:26:59.179 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:26:58 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:58 smithi188 ceph-mon[36398]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:26:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:58 smithi188 ceph-mon[36398]: osdmap e29: 4 total, 4 up, 4 in 2024-04-13T03:26:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:58 smithi188 ceph-mon[36398]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:26:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:26:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:26:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:26:58 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:00.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi016 2024-04-13T03:27:00.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:00.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:00.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi016 to 8547M 2024-04-13T03:27:00.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:00.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:00.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:00 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi016 2024-04-13T03:27:00.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:00.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:00.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi016 to 8547M 2024-04-13T03:27:00.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:00.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:00.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:00 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi016 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi016 to 8547M 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:00.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:00 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:01.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:01 smithi119 ceph-mon[35835]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-13T03:27:01.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:01 smithi119 ceph-mon[35835]: from='client.? 172.21.15.119:0/1184569604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1925505-90c0-4946-b886-25813ecd4c25"}]: dispatch 2024-04-13T03:27:01.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:01 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1925505-90c0-4946-b886-25813ecd4c25"}]: dispatch 2024-04-13T03:27:01.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:01 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1925505-90c0-4946-b886-25813ecd4c25"}]': finished 2024-04-13T03:27:01.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:01 smithi119 ceph-mon[35835]: osdmap e30: 5 total, 4 up, 5 in 2024-04-13T03:27:01.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:01.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:01 smithi016 ceph-mon[31772]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-13T03:27:01.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:01 smithi016 ceph-mon[31772]: from='client.? 172.21.15.119:0/1184569604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1925505-90c0-4946-b886-25813ecd4c25"}]: dispatch 2024-04-13T03:27:01.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:01 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1925505-90c0-4946-b886-25813ecd4c25"}]: dispatch 2024-04-13T03:27:01.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:01 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1925505-90c0-4946-b886-25813ecd4c25"}]': finished 2024-04-13T03:27:01.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:01 smithi016 ceph-mon[31772]: osdmap e30: 5 total, 4 up, 5 in 2024-04-13T03:27:01.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:01.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:01 smithi188 ceph-mon[36398]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-13T03:27:01.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:01 smithi188 ceph-mon[36398]: from='client.? 172.21.15.119:0/1184569604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1925505-90c0-4946-b886-25813ecd4c25"}]: dispatch 2024-04-13T03:27:01.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:01 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1925505-90c0-4946-b886-25813ecd4c25"}]: dispatch 2024-04-13T03:27:01.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:01 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1925505-90c0-4946-b886-25813ecd4c25"}]': finished 2024-04-13T03:27:01.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:01 smithi188 ceph-mon[36398]: osdmap e30: 5 total, 4 up, 5 in 2024-04-13T03:27:01.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:02.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:02 smithi016 ceph-mon[31772]: from='client.? 172.21.15.119:0/285107373' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:27:02.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:02 smithi188 ceph-mon[36398]: from='client.? 172.21.15.119:0/285107373' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:27:02.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:02 smithi119 ceph-mon[35835]: from='client.? 172.21.15.119:0/285107373' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:27:03.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:03 smithi016 ceph-mon[31772]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-13T03:27:03.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:03 smithi188 ceph-mon[36398]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-13T03:27:03.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:03 smithi119 ceph-mon[35835]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-13T03:27:05.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:05 smithi188 ceph-mon[36398]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-13T03:27:05.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:05 smithi119 ceph-mon[35835]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-13T03:27:05.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:05 smithi016 ceph-mon[31772]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-13T03:27:07.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:07 smithi188 ceph-mon[36398]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-13T03:27:07.744 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:07 smithi119 ceph-mon[35835]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-13T03:27:07.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:07 smithi016 ceph-mon[31772]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-13T03:27:08.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-13T03:27:08.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:08.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-13T03:27:08.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:08.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-13T03:27:08.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:09.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:09 smithi119 ceph-mon[35835]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-13T03:27:09.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:09 smithi119 ceph-mon[35835]: Deploying daemon osd.4 on smithi119 2024-04-13T03:27:09.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:09 smithi016 ceph-mon[31772]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-13T03:27:09.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:09 smithi016 ceph-mon[31772]: Deploying daemon osd.4 on smithi119 2024-04-13T03:27:09.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:09 smithi188 ceph-mon[36398]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-13T03:27:09.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:09 smithi188 ceph-mon[36398]: Deploying daemon osd.4 on smithi119 2024-04-13T03:27:11.523 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:11 smithi119 ceph-mon[35835]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:11.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:11 smithi016 ceph-mon[31772]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:11.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:11 smithi188 ceph-mon[36398]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:13.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:13 smithi119 ceph-mon[35835]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:13.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:13 smithi016 ceph-mon[31772]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:13.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:13 smithi188 ceph-mon[36398]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:14 smithi188 ceph-mon[36398]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:15.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:15.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:15.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:14 smithi119 ceph-mon[35835]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:15.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:14 smithi016 ceph-mon[31772]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:17.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:16 smithi188 ceph-mon[36398]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:17.157 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:16 smithi119 ceph-mon[35835]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:17.225 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 4 on host 'smithi119' 2024-04-13T03:27:17.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.224+0000 7f1a877fe700 1 -- 172.21.15.119:0/1947093713 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1a90062ef0 con 0x7f1a7c0a3c40 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1a7c0a3c40 msgr2=0x7f1a7c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1a7c0a3c40 0x7f1a7c0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c006fd0 tx=0x7f1a8c008040 comp rx=0 tx=0).stop 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90103da0 msgr2=0x7f1a901992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90103da0 0x7f1a901992a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000b530 tx=0x7f1a8000b840 comp rx=0 tx=0).stop 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 shutdown_connections 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1a7c0a3c40 0x7f1a7c0a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a90104740 0x7f1a901a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a90100dc0 0x7f1a90198d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 --2- 172.21.15.119:0/1947093713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a90103da0 0x7f1a901992a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 >> 172.21.15.119:0/1947093713 conn(0x7f1a900fc740 msgr2=0x7f1a901070a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 shutdown_connections 2024-04-13T03:27:17.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:27:17.226+0000 7f1a983c6700 1 -- 172.21.15.119:0/1947093713 wait complete. 2024-04-13T03:27:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:16 smithi016 ceph-mon[31772]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:17.855 DEBUG:teuthology.orchestra.run.smithi119:osd.4> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.4.service 2024-04-13T03:27:17.858 INFO:tasks.cephadm:Deploying osd.5 on smithi188 with /dev/vg_nvme/lv_3... 2024-04-13T03:27:17.858 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-13T03:27:18.105 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:27:18.481 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.481 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.481 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.481 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.481 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:18 smithi119 ceph-mon[35835]: from='osd.4 [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-13T03:27:18.481 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:18 smithi119 ceph-mon[35835]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-13T03:27:18.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:18 smithi188 ceph-mon[36398]: from='osd.4 [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-13T03:27:18.501 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:18 smithi188 ceph-mon[36398]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-13T03:27:18.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:18.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:18 smithi016 ceph-mon[31772]: from='osd.4 [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-13T03:27:18.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:18 smithi016 ceph-mon[31772]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-13T03:27:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:19 smithi188 ceph-mon[36398]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:19 smithi188 ceph-mon[36398]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-13T03:27:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:19 smithi188 ceph-mon[36398]: from='osd.4 [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:27:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:19 smithi188 ceph-mon[36398]: osdmap e31: 5 total, 4 up, 5 in 2024-04-13T03:27:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:19 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:19 smithi188 ceph-mon[36398]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:27:19.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:19 smithi119 ceph-mon[35835]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:19.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:19 smithi119 ceph-mon[35835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-13T03:27:19.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:19 smithi119 ceph-mon[35835]: from='osd.4 [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:27:19.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:19 smithi119 ceph-mon[35835]: osdmap e31: 5 total, 4 up, 5 in 2024-04-13T03:27:19.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:19 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:19.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:19 smithi119 ceph-mon[35835]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:27:19.822 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:27:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:27:19.520+0000 7f2820b00700 -1 osd.4 0 waiting for initial osdmap 2024-04-13T03:27:19.822 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:27:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:27:19.527+0000 7f281c8f3700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T03:27:19.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:19 smithi016 ceph-mon[31772]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:19.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:19 smithi016 ceph-mon[31772]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-13T03:27:19.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:19 smithi016 ceph-mon[31772]: from='osd.4 [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:27:19.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:19 smithi016 ceph-mon[31772]: osdmap e31: 5 total, 4 up, 5 in 2024-04-13T03:27:19.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:19 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:19.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:19 smithi016 ceph-mon[31772]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:27:20.247 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-13T03:27:20.247 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-13T03:27:20.247 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-13T03:27:20.247 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-13T03:27:20.247 INFO:teuthology.orchestra.run.smithi188.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134081 s, 782 MB/s 2024-04-13T03:27:20.247 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 2024-04-13T03:27:20.247 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-13T03:27:20.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:20 smithi119 ceph-mon[35835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:27:20.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:20 smithi119 ceph-mon[35835]: osdmap e32: 5 total, 4 up, 5 in 2024-04-13T03:27:20.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:20.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:20.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:20 smithi119 ceph-mon[35835]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:20.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:20 smithi188 ceph-mon[36398]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:27:20.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:20 smithi188 ceph-mon[36398]: osdmap e32: 5 total, 4 up, 5 in 2024-04-13T03:27:20.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:20.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:20.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:20 smithi188 ceph-mon[36398]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:20.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:20 smithi016 ceph-mon[31772]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:27:20.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:20 smithi016 ceph-mon[31772]: osdmap e32: 5 total, 4 up, 5 in 2024-04-13T03:27:20.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:20.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:20.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:20 smithi016 ceph-mon[31772]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-13T03:27:20.873 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2024-04-13T03:27:21.120 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: osd.4 [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] boot 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: osdmap e33: 5 total, 5 up, 5 in 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi119 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi119 to 8540M 2024-04-13T03:27:21.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:21.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:21.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:21.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:21.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:21.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:21 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:27:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:27:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: osd.4 [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] boot 2024-04-13T03:27:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: osdmap e33: 5 total, 5 up, 5 in 2024-04-13T03:27:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi119 2024-04-13T03:27:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi119 to 8540M 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:21.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:21 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: osd.4 [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] boot 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: osdmap e33: 5 total, 5 up, 5 in 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi119 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi119 to 8540M 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:21.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:21 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.628+0000 7fe332ca2700 1 -- 172.21.15.188:0/629854638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c100dc0 msgr2=0x7fe32c1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.628+0000 7fe332ca2700 1 --2- 172.21.15.188:0/629854638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c100dc0 0x7fe32c1011e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009990 tx=0x7fe31c009ca0 comp rx=0 tx=0).stop 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.628+0000 7fe332ca2700 1 -- 172.21.15.188:0/629854638 shutdown_connections 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.628+0000 7fe332ca2700 1 --2- 172.21.15.188:0/629854638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe32c104740 0x7fe32c10aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.628+0000 7fe332ca2700 1 --2- 172.21.15.188:0/629854638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c100dc0 0x7fe32c1011e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.628+0000 7fe332ca2700 1 --2- 172.21.15.188:0/629854638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe32c103da0 0x7fe32c104200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.628+0000 7fe332ca2700 1 -- 172.21.15.188:0/629854638 >> 172.21.15.188:0/629854638 conn(0x7fe32c0fc740 msgr2=0x7fe32c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.628+0000 7fe332ca2700 1 -- 172.21.15.188:0/629854638 shutdown_connections 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.628+0000 7fe332ca2700 1 -- 172.21.15.188:0/629854638 wait complete. 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe332ca2700 1 Processor -- start 2024-04-13T03:27:22.631 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe332ca2700 1 -- start start 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe332ca2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe32c100dc0 0x7fe32c198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe332ca2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c103da0 0x7fe32c1992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe332ca2700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe32c104740 0x7fe32c1a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe332ca2700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c10e730 con 0x7fe32c104740 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe332ca2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c10e5b0 con 0x7fe32c100dc0 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe332ca2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c10e8b0 con 0x7fe32c103da0 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe32bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c103da0 0x7fe32c1992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe32bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c103da0 0x7fe32c1992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:43528/0 (socket says 172.21.15.188:43528) 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe32bfff700 1 -- 172.21.15.188:0/4040658253 learned_addr learned my addr 172.21.15.188:0/4040658253 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe330a3e700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe32c100dc0 0x7fe32c198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe32bfff700 1 -- 172.21.15.188:0/4040658253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe32c100dc0 msgr2=0x7fe32c198d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe32bfff700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe32c100dc0 0x7fe32c198d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe32bfff700 1 -- 172.21.15.188:0/4040658253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe32c104740 msgr2=0x7fe32c1a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:22.632 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe32bfff700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe32c104740 0x7fe32c1a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:22.633 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe32bfff700 1 -- 172.21.15.188:0/4040658253 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32c1a09e0 con 0x7fe32c103da0 2024-04-13T03:27:22.633 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe330a3e700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe32c100dc0 0x7fe32c198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-13T03:27:22.633 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe32bfff700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c103da0 0x7fe32c1992d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c920 tx=0x7fe32000cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:27:22.633 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.629+0000 7fe329ffb700 1 -- 172.21.15.188:0/4040658253 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320007b00 con 0x7fe32c103da0 2024-04-13T03:27:22.633 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.630+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32c1a0cd0 con 0x7fe32c103da0 2024-04-13T03:27:22.633 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.630+0000 7fe329ffb700 1 -- 172.21.15.188:0/4040658253 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe320004d10 con 0x7fe32c103da0 2024-04-13T03:27:22.633 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.630+0000 7fe329ffb700 1 -- 172.21.15.188:0/4040658253 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3200056e0 con 0x7fe32c103da0 2024-04-13T03:27:22.633 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.630+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32c1a1240 con 0x7fe32c103da0 2024-04-13T03:27:22.634 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.631+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe32c0697a0 con 0x7fe32c103da0 2024-04-13T03:27:22.634 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.632+0000 7fe329ffb700 1 -- 172.21.15.188:0/4040658253 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe320007ca0 con 0x7fe32c103da0 2024-04-13T03:27:22.634 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.632+0000 7fe329ffb700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3140a3c40 0x7fe3140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:27:22.634 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.633+0000 7fe329ffb700 1 -- 172.21.15.188:0/4040658253 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fe3200ceff0 con 0x7fe32c103da0 2024-04-13T03:27:22.634 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.633+0000 7fe330a3e700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3140a3c40 0x7fe3140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:27:22.634 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.634+0000 7fe330a3e700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3140a3c40 0x7fe3140a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe31c000c00 tx=0x7fe31c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:27:22.638 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.637+0000 7fe329ffb700 1 -- 172.21.15.188:0/4040658253 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe320014030 con 0x7fe32c103da0 2024-04-13T03:27:22.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:22 smithi119 ceph-mon[35835]: osdmap e34: 5 total, 5 up, 5 in 2024-04-13T03:27:22.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:22 smithi119 ceph-mon[35835]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:22.842 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:22.841+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe32c062ef0 con 0x7fe3140a3c40 2024-04-13T03:27:22.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:22 smithi188 ceph-mon[36398]: osdmap e34: 5 total, 5 up, 5 in 2024-04-13T03:27:22.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:22 smithi188 ceph-mon[36398]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:22.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:22 smithi016 ceph-mon[31772]: osdmap e34: 5 total, 5 up, 5 in 2024-04-13T03:27:22.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:22 smithi016 ceph-mon[31772]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:23.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:23 smithi188 ceph-mon[36398]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:27:23.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:23 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:27:23.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:23 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:27:23.759 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:23 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:23 smithi119 ceph-mon[35835]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:27:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:23 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:27:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:23 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:27:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:23 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:23.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:23 smithi016 ceph-mon[31772]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:27:23.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:23 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:27:23.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:23 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:27:23.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:23 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:24 smithi188 ceph-mon[36398]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:24.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:24 smithi016 ceph-mon[31772]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:25.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:24 smithi119 ceph-mon[35835]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:25.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:25 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/3304408764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3"}]: dispatch 2024-04-13T03:27:25.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:25 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3"}]: dispatch 2024-04-13T03:27:25.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:25 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3"}]': finished 2024-04-13T03:27:25.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:25 smithi016 ceph-mon[31772]: osdmap e35: 6 total, 5 up, 6 in 2024-04-13T03:27:25.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:25.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:25 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/3304408764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3"}]: dispatch 2024-04-13T03:27:25.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:25 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3"}]: dispatch 2024-04-13T03:27:25.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:25 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3"}]': finished 2024-04-13T03:27:25.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:25 smithi188 ceph-mon[36398]: osdmap e35: 6 total, 5 up, 6 in 2024-04-13T03:27:25.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:26.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:25 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/3304408764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3"}]: dispatch 2024-04-13T03:27:26.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:25 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3"}]: dispatch 2024-04-13T03:27:26.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:25 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3"}]': finished 2024-04-13T03:27:26.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:25 smithi119 ceph-mon[35835]: osdmap e35: 6 total, 5 up, 6 in 2024-04-13T03:27:26.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:26.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:26 smithi016 ceph-mon[31772]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:26.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:26 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/1171298503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:27:26.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:26 smithi188 ceph-mon[36398]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:26.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:26 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/1171298503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:27:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:26 smithi119 ceph-mon[35835]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:26 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/1171298503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:27:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:28 smithi016 ceph-mon[31772]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:28 smithi188 ceph-mon[36398]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:29.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:28 smithi119 ceph-mon[35835]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:31.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:30 smithi016 ceph-mon[31772]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:30 smithi188 ceph-mon[36398]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:31.123 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:30 smithi119 ceph-mon[35835]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:32 smithi188 ceph-mon[36398]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:32 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-13T03:27:33.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:32 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:33.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:32 smithi119 ceph-mon[35835]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:33.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:32 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-13T03:27:33.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:32 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:33.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:32 smithi016 ceph-mon[31772]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:33.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:32 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-13T03:27:33.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:32 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:33.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:33 smithi188 ceph-mon[36398]: Deploying daemon osd.5 on smithi188 2024-04-13T03:27:34.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:33 smithi119 ceph-mon[35835]: Deploying daemon osd.5 on smithi188 2024-04-13T03:27:34.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:33 smithi016 ceph-mon[31772]: Deploying daemon osd.5 on smithi188 2024-04-13T03:27:34.895 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:34 smithi188 ceph-mon[36398]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:35.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:34 smithi119 ceph-mon[35835]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:35.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:34 smithi016 ceph-mon[31772]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:37.033 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:36 smithi188 ceph-mon[36398]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:36 smithi119 ceph-mon[35835]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:37.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:36 smithi016 ceph-mon[31772]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:39.197 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:38 smithi188 ceph-mon[36398]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:39.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:39.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:39.198 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:38 smithi119 ceph-mon[35835]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:39.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:38 smithi016 ceph-mon[31772]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:39.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:39.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:39.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:41.087 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:40 smithi188 ceph-mon[36398]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:41.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:40 smithi016 ceph-mon[31772]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:41.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:40 smithi119 ceph-mon[35835]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:41.433 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.432+0000 7fe329ffb700 1 -- 172.21.15.188:0/4040658253 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe32c062ef0 con 0x7fe3140a3c40 2024-04-13T03:27:41.433 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 5 on host 'smithi188' 2024-04-13T03:27:41.435 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.434+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3140a3c40 msgr2=0x7fe3140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:41.435 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.434+0000 7fe332ca2700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3140a3c40 0x7fe3140a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe31c000c00 tx=0x7fe31c005fb0 comp rx=0 tx=0).stop 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.434+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c103da0 msgr2=0x7fe32c1992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.434+0000 7fe332ca2700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c103da0 0x7fe32c1992d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c920 tx=0x7fe32000cce0 comp rx=0 tx=0).stop 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.435+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 shutdown_connections 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.435+0000 7fe332ca2700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3140a3c40 0x7fe3140a6100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.435+0000 7fe332ca2700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe32c104740 0x7fe32c1a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.435+0000 7fe332ca2700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe32c100dc0 0x7fe32c198d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.435+0000 7fe332ca2700 1 --2- 172.21.15.188:0/4040658253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe32c103da0 0x7fe32c1992d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.435+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 >> 172.21.15.188:0/4040658253 conn(0x7fe32c0fc740 msgr2=0x7fe32c1070a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.435+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 shutdown_connections 2024-04-13T03:27:41.436 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:27:41.435+0000 7fe332ca2700 1 -- 172.21.15.188:0/4040658253 wait complete. 2024-04-13T03:27:42.058 DEBUG:teuthology.orchestra.run.smithi188:osd.5> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.5.service 2024-04-13T03:27:42.060 INFO:tasks.cephadm:Deploying osd.6 on smithi016 with /dev/vg_nvme/lv_2... 2024-04-13T03:27:42.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-13T03:27:42.544 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:27:42.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:42 smithi016 ceph-mon[31772]: from='osd.5 [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-13T03:27:42.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:42 smithi016 ceph-mon[31772]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-13T03:27:42.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.698 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.819 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:42 smithi188 ceph-mon[36398]: from='osd.5 [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-13T03:27:42.820 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:42 smithi188 ceph-mon[36398]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-13T03:27:42.820 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.820 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:42 smithi119 ceph-mon[35835]: from='osd.5 [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-13T03:27:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:42 smithi119 ceph-mon[35835]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-13T03:27:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:43 smithi016 ceph-mon[31772]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:43 smithi016 ceph-mon[31772]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-13T03:27:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:43 smithi016 ceph-mon[31772]: osdmap e36: 6 total, 5 up, 6 in 2024-04-13T03:27:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:43 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:43 smithi016 ceph-mon[31772]: from='osd.5 [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:27:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:43 smithi016 ceph-mon[31772]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:27:43.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:43 smithi119 ceph-mon[35835]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:43.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:43 smithi119 ceph-mon[35835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-13T03:27:43.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:43 smithi119 ceph-mon[35835]: osdmap e36: 6 total, 5 up, 6 in 2024-04-13T03:27:43.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:43 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:43.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:43 smithi119 ceph-mon[35835]: from='osd.5 [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:27:43.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:43 smithi119 ceph-mon[35835]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:27:43.824 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:43 smithi188 ceph-mon[36398]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:43.824 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:43 smithi188 ceph-mon[36398]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-13T03:27:43.824 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:43 smithi188 ceph-mon[36398]: osdmap e36: 6 total, 5 up, 6 in 2024-04-13T03:27:43.824 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:43 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:43.824 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:43 smithi188 ceph-mon[36398]: from='osd.5 [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:27:43.824 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:43 smithi188 ceph-mon[36398]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:27:43.824 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:27:43 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:27:43.432+0000 7f94f5d4c700 -1 osd.5 0 waiting for initial osdmap 2024-04-13T03:27:43.824 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:27:43 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:27:43.438+0000 7f94f1b3f700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T03:27:44.768 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:44 smithi016 ceph-mon[31772]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:27:44.769 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:44 smithi016 ceph-mon[31772]: osdmap e37: 6 total, 5 up, 6 in 2024-04-13T03:27:44.769 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:44.769 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:44 smithi016 ceph-mon[31772]: osd.5 [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] boot 2024-04-13T03:27:44.769 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:44 smithi016 ceph-mon[31772]: osdmap e38: 6 total, 6 up, 6 in 2024-04-13T03:27:44.769 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:44.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:44 smithi188 ceph-mon[36398]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:27:44.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:44 smithi188 ceph-mon[36398]: osdmap e37: 6 total, 5 up, 6 in 2024-04-13T03:27:44.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:44 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:44.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:44 smithi188 ceph-mon[36398]: osd.5 [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] boot 2024-04-13T03:27:44.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:44 smithi188 ceph-mon[36398]: osdmap e38: 6 total, 6 up, 6 in 2024-04-13T03:27:44.782 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:44 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:44.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:44 smithi119 ceph-mon[35835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:27:44.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:44 smithi119 ceph-mon[35835]: osdmap e37: 6 total, 5 up, 6 in 2024-04-13T03:27:44.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:44 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:44.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:44 smithi119 ceph-mon[35835]: osd.5 [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] boot 2024-04-13T03:27:44.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:44 smithi119 ceph-mon[35835]: osdmap e38: 6 total, 6 up, 6 in 2024-04-13T03:27:44.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:44 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:27:45.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:45 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:27:45.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:45 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:27:45.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:45 smithi119 ceph-mon[35835]: pgmap v149: 1 pgs: 1 peering; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:45.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:45 smithi119 ceph-mon[35835]: osdmap e39: 6 total, 6 up, 6 in 2024-04-13T03:27:45.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:45 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:27:45.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:45 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:27:45.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:45 smithi016 ceph-mon[31772]: pgmap v149: 1 pgs: 1 peering; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:45.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:45 smithi016 ceph-mon[31772]: osdmap e39: 6 total, 6 up, 6 in 2024-04-13T03:27:45.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:45 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:27:45.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:45 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:27:45.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:45 smithi188 ceph-mon[36398]: pgmap v149: 1 pgs: 1 peering; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-13T03:27:45.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:45 smithi188 ceph-mon[36398]: osdmap e39: 6 total, 6 up, 6 in 2024-04-13T03:27:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-13T03:27:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-13T03:27:45.908 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-13T03:27:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-13T03:27:45.908 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0327142 s, 321 MB/s 2024-04-13T03:27:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-13T03:27:46.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi188 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi188 to 8540M 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: pgmap v152: 1 pgs: 1 peering; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:46.807 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:46.808 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.808 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:46 smithi016 ceph-mon[31772]: osdmap e40: 6 total, 6 up, 6 in 2024-04-13T03:27:46.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-04-13T03:27:46.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi188 2024-04-13T03:27:46.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:46.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:46.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi188 to 8540M 2024-04-13T03:27:46.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:46.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:46.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: pgmap v152: 1 pgs: 1 peering; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:27:46.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:46.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:46.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:46.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:46 smithi119 ceph-mon[35835]: osdmap e40: 6 total, 6 up, 6 in 2024-04-13T03:27:46.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi188 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi188 to 8540M 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: pgmap v152: 1 pgs: 1 peering; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:46.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:46.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:27:46.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:46 smithi188 ceph-mon[36398]: osdmap e40: 6 total, 6 up, 6 in 2024-04-13T03:27:47.285 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:27:48.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:48 smithi016 ceph-mon[31772]: osdmap e41: 6 total, 6 up, 6 in 2024-04-13T03:27:48.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:48 smithi016 ceph-mon[31772]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:27:48.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:48 smithi188 ceph-mon[36398]: osdmap e41: 6 total, 6 up, 6 in 2024-04-13T03:27:48.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:48 smithi188 ceph-mon[36398]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:27:49.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:48 smithi119 ceph-mon[35835]: osdmap e41: 6 total, 6 up, 6 in 2024-04-13T03:27:49.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:48 smithi119 ceph-mon[35835]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:27:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.666+0000 7f179d400700 1 -- 172.21.15.16:0/2112031061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1798105f50 msgr2=0x7f17981063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.666+0000 7f179d400700 1 --2- 172.21.15.16:0/2112031061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1798105f50 0x7f17981063b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1784009920 tx=0x7f1784009c30 comp rx=0 tx=0).stop 2024-04-13T03:27:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.666+0000 7f179d400700 1 -- 172.21.15.16:0/2112031061 shutdown_connections 2024-04-13T03:27:49.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.666+0000 7f179d400700 1 --2- 172.21.15.16:0/2112031061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17981068f0 0x7f179810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.666+0000 7f179d400700 1 --2- 172.21.15.16:0/2112031061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1798104d50 0x7f1798105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.666+0000 7f179d400700 1 --2- 172.21.15.16:0/2112031061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1798105f50 0x7f17981063b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.666+0000 7f179d400700 1 -- 172.21.15.16:0/2112031061 >> 172.21.15.16:0/2112031061 conn(0x7f1798100310 msgr2=0x7f1798102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:27:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.666+0000 7f179d400700 1 -- 172.21.15.16:0/2112031061 shutdown_connections 2024-04-13T03:27:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.666+0000 7f179d400700 1 -- 172.21.15.16:0/2112031061 wait complete. 2024-04-13T03:27:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.667+0000 7f179d400700 1 Processor -- start 2024-04-13T03:27:49.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.667+0000 7f179d400700 1 -- start start 2024-04-13T03:27:49.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f179d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1798104d50 0x7f179819b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:27:49.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f179d400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1798105f50 0x7f179819b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:27:49.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f179d400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17981068f0 0x7f17981a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:27:49.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f179d400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1798110960 con 0x7f1798104d50 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f179d400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f17981107e0 con 0x7f17981068f0 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f179d400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1798110ae0 con 0x7f1798105f50 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1796800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1798104d50 0x7f179819b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1796800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1798104d50 0x7f179819b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39488/0 (socket says 172.21.15.16:39488) 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1796800700 1 -- 172.21.15.16:0/2241083057 learned_addr learned my addr 172.21.15.16:0/2241083057 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1795e00700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1798105f50 0x7f179819b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1797200700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17981068f0 0x7f17981a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1796800700 1 -- 172.21.15.16:0/2241083057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1798105f50 msgr2=0x7f179819b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1796800700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1798105f50 0x7f179819b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1796800700 1 -- 172.21.15.16:0/2241083057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17981068f0 msgr2=0x7f17981a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1796800700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17981068f0 0x7f17981a25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.668+0000 7f1796800700 1 -- 172.21.15.16:0/2241083057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17981a2c70 con 0x7f1798104d50 2024-04-13T03:27:49.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.669+0000 7f1796800700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1798104d50 0x7f179819b010 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f178800e9a0 tx=0x7f178800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:27:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.669+0000 7f178f400700 1 -- 172.21.15.16:0/2241083057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788004220 con 0x7f1798104d50 2024-04-13T03:27:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.669+0000 7f178f400700 1 -- 172.21.15.16:0/2241083057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17880043c0 con 0x7f1798104d50 2024-04-13T03:27:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.669+0000 7f178f400700 1 -- 172.21.15.16:0/2241083057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788003df0 con 0x7f1798104d50 2024-04-13T03:27:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.669+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17981a2f60 con 0x7f1798104d50 2024-04-13T03:27:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.669+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17981a33c0 con 0x7f1798104d50 2024-04-13T03:27:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.670+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17980697a0 con 0x7f1798104d50 2024-04-13T03:27:49.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.673+0000 7f178f400700 1 -- 172.21.15.16:0/2241083057 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f178800c810 con 0x7f1798104d50 2024-04-13T03:27:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.674+0000 7f178f400700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f177c0a3cc0 0x7f177c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:27:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.674+0000 7f178f400700 1 -- 172.21.15.16:0/2241083057 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f17880cf700 con 0x7f1798104d50 2024-04-13T03:27:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.674+0000 7f1795e00700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f177c0a3cc0 0x7f177c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:27:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.675+0000 7f1795e00700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f177c0a3cc0 0x7f177c0a6180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f178400fca0 tx=0x7f1784019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:27:49.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.679+0000 7f178f400700 1 -- 172.21.15.16:0/2241083057 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f178809cf00 con 0x7f1798104d50 2024-04-13T03:27:49.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:49 smithi016 ceph-mon[31772]: osdmap e42: 6 total, 6 up, 6 in 2024-04-13T03:27:49.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:49 smithi188 ceph-mon[36398]: osdmap e42: 6 total, 6 up, 6 in 2024-04-13T03:27:49.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:27:49.922+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1798062ef0 con 0x7f177c0a3cc0 2024-04-13T03:27:50.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:49 smithi119 ceph-mon[35835]: osdmap e42: 6 total, 6 up, 6 in 2024-04-13T03:27:50.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:50 smithi016 ceph-mon[31772]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-13T03:27:50.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:50 smithi016 ceph-mon[31772]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:27:50.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:27:50.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:27:50.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:50.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:50 smithi188 ceph-mon[36398]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-13T03:27:50.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:50 smithi188 ceph-mon[36398]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:27:50.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:27:50.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:27:50.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:50 smithi119 ceph-mon[35835]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-13T03:27:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:50 smithi119 ceph-mon[35835]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:27:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:27:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:27:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:52 smithi016 ceph-mon[31772]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-13T03:27:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:52 smithi188 ceph-mon[36398]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-13T03:27:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:52 smithi119 ceph-mon[35835]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-13T03:27:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:54 smithi016 ceph-mon[31772]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-13T03:27:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:54 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2851773666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "facc0570-ab47-4224-9ac9-290d12b6a5f0"}]: dispatch 2024-04-13T03:27:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:54 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "facc0570-ab47-4224-9ac9-290d12b6a5f0"}]: dispatch 2024-04-13T03:27:55.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:54 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "facc0570-ab47-4224-9ac9-290d12b6a5f0"}]': finished 2024-04-13T03:27:55.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:54 smithi016 ceph-mon[31772]: osdmap e43: 7 total, 6 up, 7 in 2024-04-13T03:27:55.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:27:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:54 smithi188 ceph-mon[36398]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-13T03:27:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:54 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2851773666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "facc0570-ab47-4224-9ac9-290d12b6a5f0"}]: dispatch 2024-04-13T03:27:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:54 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "facc0570-ab47-4224-9ac9-290d12b6a5f0"}]: dispatch 2024-04-13T03:27:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:54 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "facc0570-ab47-4224-9ac9-290d12b6a5f0"}]': finished 2024-04-13T03:27:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:54 smithi188 ceph-mon[36398]: osdmap e43: 7 total, 6 up, 7 in 2024-04-13T03:27:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:27:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:54 smithi119 ceph-mon[35835]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-13T03:27:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:54 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2851773666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "facc0570-ab47-4224-9ac9-290d12b6a5f0"}]: dispatch 2024-04-13T03:27:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:54 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "facc0570-ab47-4224-9ac9-290d12b6a5f0"}]: dispatch 2024-04-13T03:27:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:54 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "facc0570-ab47-4224-9ac9-290d12b6a5f0"}]': finished 2024-04-13T03:27:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:54 smithi119 ceph-mon[35835]: osdmap e43: 7 total, 6 up, 7 in 2024-04-13T03:27:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:27:56.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:55 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/531990344' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:27:56.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:55 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/531990344' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:27:56.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:55 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/531990344' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:27:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:56 smithi188 ceph-mon[36398]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-13T03:27:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:56 smithi119 ceph-mon[35835]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-13T03:27:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:56 smithi016 ceph-mon[31772]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-13T03:27:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:27:58 smithi016 ceph-mon[31772]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-13T03:27:59.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:27:58 smithi119 ceph-mon[35835]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-13T03:27:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:27:58 smithi188 ceph-mon[36398]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-13T03:28:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:00 smithi119 ceph-mon[35835]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:00 smithi016 ceph-mon[31772]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:00 smithi188 ceph-mon[36398]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:02 smithi119 ceph-mon[35835]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:02 smithi016 ceph-mon[31772]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:02 smithi188 ceph-mon[36398]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:04.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-13T03:28:04.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:03 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:04.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:03 smithi188 ceph-mon[36398]: Deploying daemon osd.6 on smithi016 2024-04-13T03:28:04.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-13T03:28:04.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:03 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:04.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:03 smithi119 ceph-mon[35835]: Deploying daemon osd.6 on smithi016 2024-04-13T03:28:04.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-13T03:28:04.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:03 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:04.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:03 smithi016 ceph-mon[31772]: Deploying daemon osd.6 on smithi016 2024-04-13T03:28:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:04 smithi016 ceph-mon[31772]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:04 smithi119 ceph-mon[35835]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:04 smithi188 ceph-mon[36398]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:07.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:06 smithi016 ceph-mon[31772]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:06 smithi119 ceph-mon[35835]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:06 smithi188 ceph-mon[36398]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:08 smithi016 ceph-mon[31772]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:08 smithi119 ceph-mon[35835]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:08 smithi188 ceph-mon[36398]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:10 smithi119 ceph-mon[35835]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:11.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:10 smithi016 ceph-mon[31772]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:11.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:10 smithi188 ceph-mon[36398]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:13 smithi016 ceph-mon[31772]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:13 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:13.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:13 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:13.443 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:13 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:13.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:13 smithi119 ceph-mon[35835]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:13.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:13 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:13.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:13 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:13.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:13 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:13.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:13 smithi188 ceph-mon[36398]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:13.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:13 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:13.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:13 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:13.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:13 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:15.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:15 smithi016 ceph-mon[31772]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:15 smithi119 ceph-mon[35835]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:15.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:15 smithi188 ceph-mon[36398]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:16.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:16 smithi119 ceph-mon[35835]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-13T03:28:16.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:16 smithi016 ceph-mon[31772]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-13T03:28:16.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:16 smithi188 ceph-mon[36398]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-13T03:28:17.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:17 smithi016 ceph-mon[31772]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:17.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:17 smithi016 ceph-mon[31772]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-13T03:28:17.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:17 smithi016 ceph-mon[31772]: osdmap e44: 7 total, 6 up, 7 in 2024-04-13T03:28:17.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:17 smithi016 ceph-mon[31772]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:28:17.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:17.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:17.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:17 smithi119 ceph-mon[35835]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:17 smithi119 ceph-mon[35835]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-13T03:28:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:17 smithi119 ceph-mon[35835]: osdmap e44: 7 total, 6 up, 7 in 2024-04-13T03:28:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:17 smithi119 ceph-mon[35835]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:28:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:17.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.621+0000 7f178f400700 1 -- 172.21.15.16:0/2241083057 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1798062ef0 con 0x7f177c0a3cc0 2024-04-13T03:28:17.622 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 6 on host 'smithi016' 2024-04-13T03:28:17.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:17 smithi188 ceph-mon[36398]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:17.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:17 smithi188 ceph-mon[36398]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-13T03:28:17.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:17 smithi188 ceph-mon[36398]: osdmap e44: 7 total, 6 up, 7 in 2024-04-13T03:28:17.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:17 smithi188 ceph-mon[36398]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:28:17.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:17.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:17.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f177c0a3cc0 msgr2=0x7f177c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f177c0a3cc0 0x7f177c0a6180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f178400fca0 tx=0x7f1784019040 comp rx=0 tx=0).stop 2024-04-13T03:28:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1798104d50 msgr2=0x7f179819b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1798104d50 0x7f179819b010 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f178800e9a0 tx=0x7f178800ecb0 comp rx=0 tx=0).stop 2024-04-13T03:28:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 shutdown_connections 2024-04-13T03:28:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f177c0a3cc0 0x7f177c0a6180 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17981068f0 0x7f17981a25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1798104d50 0x7f179819b010 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 --2- 172.21.15.16:0/2241083057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1798105f50 0x7f179819b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.625+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 >> 172.21.15.16:0/2241083057 conn(0x7f1798100310 msgr2=0x7f1798109370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:28:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.626+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 shutdown_connections 2024-04-13T03:28:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:28:17.626+0000 7f179d400700 1 -- 172.21.15.16:0/2241083057 wait complete. 2024-04-13T03:28:18.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:18 smithi016 ceph-mon[31772]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:28:18.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:18 smithi016 ceph-mon[31772]: osdmap e45: 7 total, 6 up, 7 in 2024-04-13T03:28:18.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:18.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:18.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:18.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:18.489 DEBUG:teuthology.orchestra.run.smithi016:osd.6> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.6.service 2024-04-13T03:28:18.495 INFO:tasks.cephadm:Deploying osd.7 on smithi119 with /dev/vg_nvme/lv_2... 2024-04-13T03:28:18.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-13T03:28:18.524 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:18 smithi119 ceph-mon[35835]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:28:18.525 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:18 smithi119 ceph-mon[35835]: osdmap e45: 7 total, 6 up, 7 in 2024-04-13T03:28:18.525 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:18.525 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:18.525 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:18.525 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:18.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:18 smithi188 ceph-mon[36398]: from='osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:28:18.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:18 smithi188 ceph-mon[36398]: osdmap e45: 7 total, 6 up, 7 in 2024-04-13T03:28:18.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:18.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:18.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:18.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:18.740 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:28:19.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:19 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:28:19.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:19 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:28:19.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:19 smithi016 ceph-mon[31772]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:19.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:19 smithi016 ceph-mon[31772]: osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] boot 2024-04-13T03:28:19.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:19 smithi016 ceph-mon[31772]: osdmap e46: 7 total, 7 up, 7 in 2024-04-13T03:28:19.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:19 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:19 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:28:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:19 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:28:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:19 smithi119 ceph-mon[35835]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:19 smithi119 ceph-mon[35835]: osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] boot 2024-04-13T03:28:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:19 smithi119 ceph-mon[35835]: osdmap e46: 7 total, 7 up, 7 in 2024-04-13T03:28:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:19 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:19.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:19 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:28:19.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:19 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:28:19.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:19 smithi188 ceph-mon[36398]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-13T03:28:19.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:19 smithi188 ceph-mon[36398]: osd.6 [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] boot 2024-04-13T03:28:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:19 smithi188 ceph-mon[36398]: osdmap e46: 7 total, 7 up, 7 in 2024-04-13T03:28:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:19 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:28:20.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:20 smithi119 ceph-mon[35835]: osdmap e47: 7 total, 7 up, 7 in 2024-04-13T03:28:20.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:20 smithi016 ceph-mon[31772]: osdmap e47: 7 total, 7 up, 7 in 2024-04-13T03:28:20.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:20 smithi188 ceph-mon[36398]: osdmap e47: 7 total, 7 up, 7 in 2024-04-13T03:28:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-13T03:28:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-13T03:28:20.706 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-04-13T03:28:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-04-13T03:28:20.706 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114987 s, 912 MB/s 2024-04-13T03:28:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-04-13T03:28:21.411 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:21 smithi119 ceph-mon[35835]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:21.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_2 2024-04-13T03:28:21.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:21 smithi016 ceph-mon[31772]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:21.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:21 smithi188 ceph-mon[36398]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:28:23.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 -- 172.21.15.119:0/1725140283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c102b10 msgr2=0x7f2c9c102f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:23.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/1725140283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c102b10 0x7f2c9c102f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2c900099c0 tx=0x7f2c90009cd0 comp rx=0 tx=0).stop 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 -- 172.21.15.119:0/1725140283 shutdown_connections 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/1725140283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c1046b0 0x7f2c9c10af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/1725140283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c102b10 0x7f2c9c102f30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/1725140283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c9c103d10 0x7f2c9c104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 -- 172.21.15.119:0/1725140283 >> 172.21.15.119:0/1725140283 conn(0x7f2c9c0fe0b0 msgr2=0x7f2c9c1004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 -- 172.21.15.119:0/1725140283 shutdown_connections 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 -- 172.21.15.119:0/1725140283 wait complete. 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 Processor -- start 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 -- start start 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c102b10 0x7f2c9c076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c9c103d10 0x7f2c9c074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:28:23.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c1046b0 0x7f2c9c0753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c10e750 con 0x7f2c9c102b10 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c10e5d0 con 0x7f2c9c1046b0 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.340+0000 7f2ca2d94700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c10e8d0 con 0x7f2c9c103d10 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca1331700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c1046b0 0x7f2c9c0753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca1331700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c1046b0 0x7f2c9c0753e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50590/0 (socket says 172.21.15.119:50590) 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca1331700 1 -- 172.21.15.119:0/2316294219 learned_addr learned my addr 172.21.15.119:0/2316294219 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2c9bfff700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c9c103d10 0x7f2c9c074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca1331700 1 -- 172.21.15.119:0/2316294219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c9c103d10 msgr2=0x7f2c9c074ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca1331700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c9c103d10 0x7f2c9c074ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca1331700 1 -- 172.21.15.119:0/2316294219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c102b10 msgr2=0x7f2c9c076850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca1331700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c102b10 0x7f2c9c076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca1331700 1 -- 172.21.15.119:0/2316294219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c9c075c90 con 0x7f2c9c1046b0 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2c9bfff700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c9c103d10 0x7f2c9c074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:28:23.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca1331700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c1046b0 0x7f2c9c0753e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400b970 tx=0x7f2c9400bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:28:23.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2c99ffb700 1 -- 172.21.15.119:0/2316294219 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c94019070 con 0x7f2c9c1046b0 2024-04-13T03:28:23.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2c99ffb700 1 -- 172.21.15.119:0/2316294219 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c94014440 con 0x7f2c9c1046b0 2024-04-13T03:28:23.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c9c1a2ea0 con 0x7f2c9c1046b0 2024-04-13T03:28:23.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2c99ffb700 1 -- 172.21.15.119:0/2316294219 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c940075d0 con 0x7f2c9c1046b0 2024-04-13T03:28:23.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.341+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c9c1a3380 con 0x7f2c9c1046b0 2024-04-13T03:28:23.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.342+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c9c0697a0 con 0x7f2c9c1046b0 2024-04-13T03:28:23.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.343+0000 7f2c99ffb700 1 -- 172.21.15.119:0/2316294219 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2c9401c070 con 0x7f2c9c1046b0 2024-04-13T03:28:23.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.344+0000 7f2c99ffb700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c8c0a3c40 0x7f2c8c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:28:23.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.344+0000 7f2c99ffb700 1 -- 172.21.15.119:0/2316294219 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f2c940ceb00 con 0x7f2c9c1046b0 2024-04-13T03:28:23.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.344+0000 7f2ca0b30700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c8c0a3c40 0x7f2c8c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:28:23.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.345+0000 7f2ca0b30700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c8c0a3c40 0x7f2c8c0a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2c900098e0 tx=0x7f2c90000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:28:23.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.348+0000 7f2c99ffb700 1 -- 172.21.15.119:0/2316294219 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c9409c1f0 con 0x7f2c9c1046b0 2024-04-13T03:28:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:23.552+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2c9c062ef0 con 0x7f2c8c0a3c40 2024-04-13T03:28:23.553 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:23 smithi119 ceph-mon[35835]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:23.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:23 smithi016 ceph-mon[31772]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:23.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:23 smithi188 ceph-mon[36398]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:24.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:24 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:28:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:28:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:24.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:24.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:24.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:24.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:24.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:24 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:28:24.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:28:24.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:24.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:24.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:24.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:24.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:24.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:24.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:24.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:24 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:25.555 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:25 smithi119 ceph-mon[35835]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:28:25.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:25 smithi119 ceph-mon[35835]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:25.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:25 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi016 2024-04-13T03:28:25.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:25 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi016 to 5698M 2024-04-13T03:28:25.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:25.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:25.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:25.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:25.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:25 smithi016 ceph-mon[31772]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:28:25.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:25 smithi016 ceph-mon[31772]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:25.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:25 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi016 2024-04-13T03:28:25.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:25 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi016 to 5698M 2024-04-13T03:28:25.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:25.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:25.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:25.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:25 smithi188 ceph-mon[36398]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:28:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:25 smithi188 ceph-mon[36398]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:25 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi016 2024-04-13T03:28:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:25 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi016 to 5698M 2024-04-13T03:28:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:27 smithi119 ceph-mon[35835]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:27 smithi119 ceph-mon[35835]: from='client.? 172.21.15.119:0/219104866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61259d55-32c3-438a-bdac-0daf88896936"}]: dispatch 2024-04-13T03:28:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:27 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61259d55-32c3-438a-bdac-0daf88896936"}]: dispatch 2024-04-13T03:28:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:27 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61259d55-32c3-438a-bdac-0daf88896936"}]': finished 2024-04-13T03:28:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:27 smithi119 ceph-mon[35835]: osdmap e48: 8 total, 7 up, 8 in 2024-04-13T03:28:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:27 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:27 smithi119 ceph-mon[35835]: from='client.? 172.21.15.119:0/3705599963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:28:27.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:27 smithi016 ceph-mon[31772]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:27.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:27 smithi016 ceph-mon[31772]: from='client.? 172.21.15.119:0/219104866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61259d55-32c3-438a-bdac-0daf88896936"}]: dispatch 2024-04-13T03:28:27.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:27 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61259d55-32c3-438a-bdac-0daf88896936"}]: dispatch 2024-04-13T03:28:27.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:27 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61259d55-32c3-438a-bdac-0daf88896936"}]': finished 2024-04-13T03:28:27.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:27 smithi016 ceph-mon[31772]: osdmap e48: 8 total, 7 up, 8 in 2024-04-13T03:28:27.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:27 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:27.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:27 smithi016 ceph-mon[31772]: from='client.? 172.21.15.119:0/3705599963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:28:27.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:27 smithi188 ceph-mon[36398]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:27.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:27 smithi188 ceph-mon[36398]: from='client.? 172.21.15.119:0/219104866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61259d55-32c3-438a-bdac-0daf88896936"}]: dispatch 2024-04-13T03:28:27.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:27 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61259d55-32c3-438a-bdac-0daf88896936"}]: dispatch 2024-04-13T03:28:27.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:27 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61259d55-32c3-438a-bdac-0daf88896936"}]': finished 2024-04-13T03:28:27.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:27 smithi188 ceph-mon[36398]: osdmap e48: 8 total, 7 up, 8 in 2024-04-13T03:28:27.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:27 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:27.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:27 smithi188 ceph-mon[36398]: from='client.? 172.21.15.119:0/3705599963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:28:29.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:29 smithi119 ceph-mon[35835]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:29.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:29 smithi016 ceph-mon[31772]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:29.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:29 smithi188 ceph-mon[36398]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:31.508 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:31 smithi119 ceph-mon[35835]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:31.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:31 smithi016 ceph-mon[31772]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:31.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:31 smithi188 ceph-mon[36398]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:33.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:33 smithi119 ceph-mon[35835]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:33.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:33 smithi016 ceph-mon[31772]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:33.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:33 smithi188 ceph-mon[36398]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:34.556 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:34 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-13T03:28:34.557 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:34 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:34.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:34 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-13T03:28:34.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:34 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:34.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:34 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-13T03:28:34.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:34 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:35.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:35 smithi119 ceph-mon[35835]: Deploying daemon osd.7 on smithi119 2024-04-13T03:28:35.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:35 smithi119 ceph-mon[35835]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:35.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:35 smithi016 ceph-mon[31772]: Deploying daemon osd.7 on smithi119 2024-04-13T03:28:35.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:35 smithi016 ceph-mon[31772]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:35.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:35 smithi188 ceph-mon[36398]: Deploying daemon osd.7 on smithi119 2024-04-13T03:28:35.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:35 smithi188 ceph-mon[36398]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:37.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:37 smithi119 ceph-mon[35835]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:37.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:37 smithi016 ceph-mon[31772]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:37.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:37 smithi188 ceph-mon[36398]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:39.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:39 smithi119 ceph-mon[35835]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:39.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:39 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:39.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:39 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:39 smithi016 ceph-mon[31772]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:39.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:39.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:39 smithi188 ceph-mon[36398]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:39.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:39 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:39.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:39 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:40.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:40.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:40.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:41.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:41 smithi119 ceph-mon[35835]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:41.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:41 smithi016 ceph-mon[31772]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:41.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:41 smithi188 ceph-mon[36398]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:42.895 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 7 on host 'smithi119' 2024-04-13T03:28:42.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.893+0000 7f2c99ffb700 1 -- 172.21.15.119:0/2316294219 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2c9c062ef0 con 0x7f2c8c0a3c40 2024-04-13T03:28:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c8c0a3c40 msgr2=0x7f2c8c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c8c0a3c40 0x7f2c8c0a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2c900098e0 tx=0x7f2c90000bc0 comp rx=0 tx=0).stop 2024-04-13T03:28:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c1046b0 msgr2=0x7f2c9c0753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c1046b0 0x7f2c9c0753e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400b970 tx=0x7f2c9400bd30 comp rx=0 tx=0).stop 2024-04-13T03:28:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 shutdown_connections 2024-04-13T03:28:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c8c0a3c40 0x7f2c8c0a6100 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c1046b0 0x7f2c9c0753e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c102b10 0x7f2c9c076850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 --2- 172.21.15.119:0/2316294219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c9c103d10 0x7f2c9c074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:42.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 >> 172.21.15.119:0/2316294219 conn(0x7f2c9c0fe0b0 msgr2=0x7f2c9c107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:28:42.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.896+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 shutdown_connections 2024-04-13T03:28:42.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:28:42.897+0000 7f2ca2d94700 1 -- 172.21.15.119:0/2316294219 wait complete. 2024-04-13T03:28:43.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:43 smithi119 ceph-mon[35835]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:43.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:43 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:43.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:43 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:43.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:43 smithi119 ceph-mon[35835]: from='osd.7 [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-13T03:28:43.488 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:43 smithi119 ceph-mon[35835]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-13T03:28:43.514 DEBUG:teuthology.orchestra.run.smithi119:osd.7> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.7.service 2024-04-13T03:28:43.516 INFO:tasks.cephadm:Deploying osd.8 on smithi188 with /dev/vg_nvme/lv_2... 2024-04-13T03:28:43.516 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-13T03:28:43.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:43 smithi016 ceph-mon[31772]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:43.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:43 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:43 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:43 smithi016 ceph-mon[31772]: from='osd.7 [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-13T03:28:43.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:43 smithi016 ceph-mon[31772]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-13T03:28:43.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:43 smithi188 ceph-mon[36398]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:43.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:43 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:43.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:43 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:43.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:43 smithi188 ceph-mon[36398]: from='osd.7 [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-13T03:28:43.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:43 smithi188 ceph-mon[36398]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-13T03:28:43.764 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:28:45.001 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:44 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:45.001 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:44 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:45.001 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:44 smithi188 ceph-mon[36398]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:45.001 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:44 smithi188 ceph-mon[36398]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-13T03:28:45.001 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:44 smithi188 ceph-mon[36398]: from='osd.7 [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:28:45.001 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:44 smithi188 ceph-mon[36398]: osdmap e49: 8 total, 7 up, 8 in 2024-04-13T03:28:45.001 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:44 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:45.001 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:44 smithi188 ceph-mon[36398]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:28:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:44 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:44 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:44 smithi119 ceph-mon[35835]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:44 smithi119 ceph-mon[35835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-13T03:28:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:44 smithi119 ceph-mon[35835]: from='osd.7 [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:28:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:44 smithi119 ceph-mon[35835]: osdmap e49: 8 total, 7 up, 8 in 2024-04-13T03:28:45.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:44 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:45.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:44 smithi119 ceph-mon[35835]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:28:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:45.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:45.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:44 smithi016 ceph-mon[31772]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:45.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:44 smithi016 ceph-mon[31772]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-13T03:28:45.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:44 smithi016 ceph-mon[31772]: from='osd.7 [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:28:45.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:44 smithi016 ceph-mon[31772]: osdmap e49: 8 total, 7 up, 8 in 2024-04-13T03:28:45.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:45.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:44 smithi016 ceph-mon[31772]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:28:45.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:28:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:28:45.299+0000 7f3b264df700 -1 osd.7 0 waiting for initial osdmap 2024-04-13T03:28:45.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:28:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:28:45.310+0000 7f3b1dac9700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T03:28:45.713 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-13T03:28:45.713 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-13T03:28:45.713 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-13T03:28:45.713 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-13T03:28:45.713 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112414 s, 933 MB/s 2024-04-13T03:28:45.713 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-13T03:28:46.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:45 smithi188 ceph-mon[36398]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:28:46.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:45 smithi188 ceph-mon[36398]: osdmap e50: 8 total, 7 up, 8 in 2024-04-13T03:28:46.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:45 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:46.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:45 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:46.046 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:45 smithi188 ceph-mon[36398]: from='osd.7 ' entity='osd.7' 2024-04-13T03:28:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:45 smithi119 ceph-mon[35835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:28:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:45 smithi119 ceph-mon[35835]: osdmap e50: 8 total, 7 up, 8 in 2024-04-13T03:28:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:45 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:45 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:45 smithi119 ceph-mon[35835]: from='osd.7 ' entity='osd.7' 2024-04-13T03:28:46.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:45 smithi016 ceph-mon[31772]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:28:46.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:45 smithi016 ceph-mon[31772]: osdmap e50: 8 total, 7 up, 8 in 2024-04-13T03:28:46.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:45 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:46.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:45 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:46.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:45 smithi016 ceph-mon[31772]: from='osd.7 ' entity='osd.7' 2024-04-13T03:28:46.359 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2024-04-13T03:28:46.604 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:28:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:46 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:28:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:46 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:28:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:46 smithi119 ceph-mon[35835]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:46 smithi119 ceph-mon[35835]: osd.7 [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] boot 2024-04-13T03:28:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:46 smithi119 ceph-mon[35835]: osdmap e51: 8 total, 8 up, 8 in 2024-04-13T03:28:47.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:46 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:46 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:28:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:46 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:28:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:46 smithi016 ceph-mon[31772]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:46 smithi016 ceph-mon[31772]: osd.7 [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] boot 2024-04-13T03:28:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:46 smithi016 ceph-mon[31772]: osdmap e51: 8 total, 8 up, 8 in 2024-04-13T03:28:47.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:46 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:46 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:28:47.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:46 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:28:47.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:46 smithi188 ceph-mon[36398]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-13T03:28:47.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:47.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:46 smithi188 ceph-mon[36398]: osd.7 [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] boot 2024-04-13T03:28:47.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:46 smithi188 ceph-mon[36398]: osdmap e51: 8 total, 8 up, 8 in 2024-04-13T03:28:47.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:46 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:28:47.991 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: osdmap e52: 8 total, 8 up, 8 in 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi119 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi119 to 5693M 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:47.992 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:47.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:47.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:47.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:47.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:47.993 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:47 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 -- 172.21.15.188:0/607925355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948102630 msgr2=0x7f9948109ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 --2- 172.21.15.188:0/607925355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948102630 0x7f9948109ef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9940009990 tx=0x7f9940009ca0 comp rx=0 tx=0).stop 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 -- 172.21.15.188:0/607925355 shutdown_connections 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 --2- 172.21.15.188:0/607925355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948102630 0x7f9948109ef0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 --2- 172.21.15.188:0/607925355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9948102fe0 0x7f9948103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 --2- 172.21.15.188:0/607925355 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9948101c90 0x7f99481020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 -- 172.21.15.188:0/607925355 >> 172.21.15.188:0/607925355 conn(0x7f99480fc740 msgr2=0x7f99480feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 -- 172.21.15.188:0/607925355 shutdown_connections 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 -- 172.21.15.188:0/607925355 wait complete. 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 Processor -- start 2024-04-13T03:28:48.198 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.195+0000 7f994efcf700 1 -- start start 2024-04-13T03:28:48.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994efcf700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948101c90 0x7f9948198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:28:48.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994efcf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9948102630 0x7f99481992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:28:48.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994efcf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9948102fe0 0x7f99481a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:28:48.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994efcf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f994810d6a0 con 0x7f9948102630 2024-04-13T03:28:48.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994efcf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f994810d520 con 0x7f9948102fe0 2024-04-13T03:28:48.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994efcf700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f994810d820 con 0x7f9948101c90 2024-04-13T03:28:48.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994cd6b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948101c90 0x7f9948198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:28:48.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994cd6b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948101c90 0x7f9948198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:35498/0 (socket says 172.21.15.188:35498) 2024-04-13T03:28:48.199 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994cd6b700 1 -- 172.21.15.188:0/384940172 learned_addr learned my addr 172.21.15.188:0/384940172 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994d56c700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9948102fe0 0x7f99481a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994cd6b700 1 -- 172.21.15.188:0/384940172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9948102fe0 msgr2=0x7f99481a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994cd6b700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9948102fe0 0x7f99481a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994cd6b700 1 -- 172.21.15.188:0/384940172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9948102630 msgr2=0x7f99481992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994cd6b700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9948102630 0x7f99481992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994cd6b700 1 -- 172.21.15.188:0/384940172 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99481a09e0 con 0x7f9948101c90 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994cd6b700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948101c90 0x7f9948198d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f993800b9c0 tx=0x7f993800bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f9945ffb700 1 -- 172.21.15.188:0/384940172 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9938019070 con 0x7f9948101c90 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99481a0cd0 con 0x7f9948101c90 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f9945ffb700 1 -- 172.21.15.188:0/384940172 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9938014440 con 0x7f9948101c90 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.196+0000 7f9945ffb700 1 -- 172.21.15.188:0/384940172 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99380125a0 con 0x7f9948101c90 2024-04-13T03:28:48.200 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.197+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99481a11c0 con 0x7f9948101c90 2024-04-13T03:28:48.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.198+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99480697a0 con 0x7f9948101c90 2024-04-13T03:28:48.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.198+0000 7f9945ffb700 1 -- 172.21.15.188:0/384940172 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f993801c070 con 0x7f9948101c90 2024-04-13T03:28:48.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.199+0000 7f9945ffb700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f99340a3c40 0x7f99340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:28:48.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.199+0000 7f9945ffb700 1 -- 172.21.15.188:0/384940172 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f99380ced00 con 0x7f9948101c90 2024-04-13T03:28:48.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.200+0000 7f9947fff700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f99340a3c40 0x7f99340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:28:48.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.200+0000 7f9947fff700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f99340a3c40 0x7f99340a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9930005950 tx=0x7f99300058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:28:48.204 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.204+0000 7f9945ffb700 1 -- 172.21.15.188:0/384940172 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f993809c220 con 0x7f9948101c90 2024-04-13T03:28:48.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: osdmap e52: 8 total, 8 up, 8 in 2024-04-13T03:28:48.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi119 2024-04-13T03:28:48.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi119 to 5693M 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:48.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:47 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: osdmap e52: 8 total, 8 up, 8 in 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi119 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi119 to 5693M 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:48.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:48.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:47 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:28:48.412 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:28:48.411+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9948101290 con 0x7f99340a3c40 2024-04-13T03:28:49.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:48 smithi188 ceph-mon[36398]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:49.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:48 smithi188 ceph-mon[36398]: osdmap e53: 8 total, 8 up, 8 in 2024-04-13T03:28:49.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:48 smithi188 ceph-mon[36398]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:28:49.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:28:49.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:28:49.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:48 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:48 smithi119 ceph-mon[35835]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:48 smithi119 ceph-mon[35835]: osdmap e53: 8 total, 8 up, 8 in 2024-04-13T03:28:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:48 smithi119 ceph-mon[35835]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:28:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:48 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:28:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:48 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:28:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:48 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:48 smithi016 ceph-mon[31772]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:48 smithi016 ceph-mon[31772]: osdmap e53: 8 total, 8 up, 8 in 2024-04-13T03:28:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:48 smithi016 ceph-mon[31772]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:28:49.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:28:49.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:28:49.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:48 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:50.031 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:49 smithi188 ceph-mon[36398]: osdmap e54: 8 total, 8 up, 8 in 2024-04-13T03:28:50.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:49 smithi119 ceph-mon[35835]: osdmap e54: 8 total, 8 up, 8 in 2024-04-13T03:28:50.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:49 smithi016 ceph-mon[31772]: osdmap e54: 8 total, 8 up, 8 in 2024-04-13T03:28:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:50 smithi119 ceph-mon[35835]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:50 smithi119 ceph-mon[35835]: osdmap e55: 8 total, 8 up, 8 in 2024-04-13T03:28:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:50 smithi016 ceph-mon[31772]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:50 smithi016 ceph-mon[31772]: osdmap e55: 8 total, 8 up, 8 in 2024-04-13T03:28:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:50 smithi188 ceph-mon[36398]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:50 smithi188 ceph-mon[36398]: osdmap e55: 8 total, 8 up, 8 in 2024-04-13T03:28:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:51 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/3592980700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59686f85-07f5-4b22-a3a6-85a29cacda2d"}]: dispatch 2024-04-13T03:28:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:51 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59686f85-07f5-4b22-a3a6-85a29cacda2d"}]: dispatch 2024-04-13T03:28:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:51 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59686f85-07f5-4b22-a3a6-85a29cacda2d"}]': finished 2024-04-13T03:28:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:51 smithi119 ceph-mon[35835]: osdmap e56: 9 total, 8 up, 9 in 2024-04-13T03:28:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:28:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:51 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/4286009209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:28:52.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:51 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/3592980700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59686f85-07f5-4b22-a3a6-85a29cacda2d"}]: dispatch 2024-04-13T03:28:52.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:51 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59686f85-07f5-4b22-a3a6-85a29cacda2d"}]: dispatch 2024-04-13T03:28:52.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:51 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59686f85-07f5-4b22-a3a6-85a29cacda2d"}]': finished 2024-04-13T03:28:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:51 smithi016 ceph-mon[31772]: osdmap e56: 9 total, 8 up, 9 in 2024-04-13T03:28:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:28:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:51 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/4286009209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:28:52.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:51 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/3592980700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59686f85-07f5-4b22-a3a6-85a29cacda2d"}]: dispatch 2024-04-13T03:28:52.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:51 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59686f85-07f5-4b22-a3a6-85a29cacda2d"}]: dispatch 2024-04-13T03:28:52.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:51 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59686f85-07f5-4b22-a3a6-85a29cacda2d"}]': finished 2024-04-13T03:28:52.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:51 smithi188 ceph-mon[36398]: osdmap e56: 9 total, 8 up, 9 in 2024-04-13T03:28:52.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:28:52.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:51 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/4286009209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:28:53.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:53 smithi119 ceph-mon[35835]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:53.636 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:53 smithi188 ceph-mon[36398]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:53.636 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:53 smithi016 ceph-mon[31772]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:55.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:55 smithi188 ceph-mon[36398]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:55 smithi119 ceph-mon[35835]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:55.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:55 smithi016 ceph-mon[31772]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:57.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:57 smithi188 ceph-mon[36398]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:57.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:57 smithi119 ceph-mon[35835]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:57.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:57 smithi016 ceph-mon[31772]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:59 smithi188 ceph-mon[36398]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:59 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-13T03:28:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:59 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:59.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:28:59 smithi188 ceph-mon[36398]: Deploying daemon osd.8 on smithi188 2024-04-13T03:28:59.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:59 smithi119 ceph-mon[35835]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:59.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:59 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-13T03:28:59.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:59 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:59.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:28:59 smithi119 ceph-mon[35835]: Deploying daemon osd.8 on smithi188 2024-04-13T03:28:59.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:59 smithi016 ceph-mon[31772]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:28:59.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-13T03:28:59.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:59 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:59.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:28:59 smithi016 ceph-mon[31772]: Deploying daemon osd.8 on smithi188 2024-04-13T03:29:01.370 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:01 smithi188 ceph-mon[36398]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:01.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:01 smithi119 ceph-mon[35835]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:01.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:01 smithi016 ceph-mon[31772]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:03.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:03 smithi188 ceph-mon[36398]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:03.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:03 smithi119 ceph-mon[35835]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:03.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:03 smithi016 ceph-mon[31772]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:05 smithi119 ceph-mon[35835]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:05 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:05 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:05 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:05 smithi016 ceph-mon[31772]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:05 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:05 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:05 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:05 smithi188 ceph-mon[36398]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:05 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:05 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:05.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:05 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:07 smithi188 ceph-mon[36398]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:07.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:07 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:07.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:07 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:07.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:07 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:07.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:07 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:07 smithi119 ceph-mon[35835]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:07 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:07 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:07 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:07 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:07 smithi016 ceph-mon[31772]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:07.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:07.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:07.902 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 8 on host 'smithi188' 2024-04-13T03:29:07.902 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.901+0000 7f9945ffb700 1 -- 172.21.15.188:0/384940172 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9948101290 con 0x7f99340a3c40 2024-04-13T03:29:07.904 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.903+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f99340a3c40 msgr2=0x7f99340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:07.904 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.903+0000 7f994efcf700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f99340a3c40 0x7f99340a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9930005950 tx=0x7f99300058e0 comp rx=0 tx=0).stop 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.903+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948101c90 msgr2=0x7f9948198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.903+0000 7f994efcf700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948101c90 0x7f9948198d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f993800b9c0 tx=0x7f993800bd80 comp rx=0 tx=0).stop 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.904+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 shutdown_connections 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.904+0000 7f994efcf700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f99340a3c40 0x7f99340a6100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.904+0000 7f994efcf700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9948102fe0 0x7f99481a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.904+0000 7f994efcf700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9948101c90 0x7f9948198d90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.904+0000 7f994efcf700 1 --2- 172.21.15.188:0/384940172 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9948102630 0x7f99481992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.904+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 >> 172.21.15.188:0/384940172 conn(0x7f99480fc740 msgr2=0x7f9948100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.904+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 shutdown_connections 2024-04-13T03:29:07.905 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:29:07.904+0000 7f994efcf700 1 -- 172.21.15.188:0/384940172 wait complete. 2024-04-13T03:29:08.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:08.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:08.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:08.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:08.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:08 smithi188 ceph-mon[36398]: from='osd.8 [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-13T03:29:08.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:08 smithi188 ceph-mon[36398]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-13T03:29:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:08 smithi119 ceph-mon[35835]: from='osd.8 [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-13T03:29:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:08 smithi119 ceph-mon[35835]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-13T03:29:08.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:08.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:08.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:08.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:08.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:08 smithi016 ceph-mon[31772]: from='osd.8 [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-13T03:29:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:08 smithi016 ceph-mon[31772]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-13T03:29:08.525 DEBUG:teuthology.orchestra.run.smithi188:osd.8> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.8.service 2024-04-13T03:29:08.528 INFO:tasks.cephadm:Deploying osd.9 on smithi016 with /dev/vg_nvme/lv_1... 2024-04-13T03:29:08.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-13T03:29:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:29:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:09 smithi188 ceph-mon[36398]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:09 smithi188 ceph-mon[36398]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-13T03:29:09.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:09 smithi188 ceph-mon[36398]: from='osd.8 [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:29:09.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:09 smithi188 ceph-mon[36398]: osdmap e57: 9 total, 8 up, 9 in 2024-04-13T03:29:09.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:09 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:29:09.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:09 smithi188 ceph-mon[36398]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:29:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:09 smithi119 ceph-mon[35835]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:09 smithi119 ceph-mon[35835]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-13T03:29:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:09 smithi119 ceph-mon[35835]: from='osd.8 [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:29:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:09 smithi119 ceph-mon[35835]: osdmap e57: 9 total, 8 up, 9 in 2024-04-13T03:29:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:09 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:29:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:09 smithi119 ceph-mon[35835]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:29:09.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:09 smithi016 ceph-mon[31772]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:09.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:09 smithi016 ceph-mon[31772]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-13T03:29:09.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:09 smithi016 ceph-mon[31772]: from='osd.8 [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:29:09.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:09 smithi016 ceph-mon[31772]: osdmap e57: 9 total, 8 up, 9 in 2024-04-13T03:29:09.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:09 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:29:09.324 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:09 smithi016 ceph-mon[31772]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:29:10.373 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:29:10 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:29:10.265+0000 7f53c334e700 -1 osd.8 0 waiting for initial osdmap 2024-04-13T03:29:10.373 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:29:10 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:29:10.276+0000 7f53ba938700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T03:29:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:10 smithi016 ceph-mon[31772]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:10 smithi016 ceph-mon[31772]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:29:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:10 smithi016 ceph-mon[31772]: osdmap e58: 9 total, 8 up, 9 in 2024-04-13T03:29:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:29:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:10 smithi016 ceph-mon[31772]: from='osd.8 ' entity='osd.8' 2024-04-13T03:29:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:10 smithi188 ceph-mon[36398]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:10 smithi188 ceph-mon[36398]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:29:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:10 smithi188 ceph-mon[36398]: osdmap e58: 9 total, 8 up, 9 in 2024-04-13T03:29:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:10 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:29:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:10 smithi188 ceph-mon[36398]: from='osd.8 ' entity='osd.8' 2024-04-13T03:29:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:10 smithi119 ceph-mon[35835]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-13T03:29:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:10 smithi119 ceph-mon[35835]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:29:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:10 smithi119 ceph-mon[35835]: osdmap e58: 9 total, 8 up, 9 in 2024-04-13T03:29:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:10 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:29:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:10 smithi119 ceph-mon[35835]: from='osd.8 ' entity='osd.8' 2024-04-13T03:29:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-13T03:29:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-13T03:29:12.104 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-13T03:29:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-13T03:29:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0635154 s, 165 MB/s 2024-04-13T03:29:12.104 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-13T03:29:12.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: osd.8 [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] boot 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: osdmap e59: 9 total, 9 up, 9 in 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:12.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:12.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:11 smithi016 ceph-mon[31772]: osdmap e60: 9 total, 9 up, 9 in 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: osd.8 [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] boot 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: osdmap e59: 9 total, 9 up, 9 in 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:12.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:12.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:11 smithi188 ceph-mon[36398]: osdmap e60: 9 total, 9 up, 9 in 2024-04-13T03:29:12.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:29:12.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:29:12.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: osd.8 [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] boot 2024-04-13T03:29:12.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: osdmap e59: 9 total, 9 up, 9 in 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:12.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:11 smithi119 ceph-mon[35835]: osdmap e60: 9 total, 9 up, 9 in 2024-04-13T03:29:13.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:12 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi188 2024-04-13T03:29:13.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:12 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi188 to 5693M 2024-04-13T03:29:13.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:12 smithi016 ceph-mon[31772]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:13.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:12 smithi016 ceph-mon[31772]: osdmap e61: 9 total, 9 up, 9 in 2024-04-13T03:29:13.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-04-13T03:29:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:12 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi188 2024-04-13T03:29:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:12 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi188 to 5693M 2024-04-13T03:29:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:12 smithi119 ceph-mon[35835]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:12 smithi119 ceph-mon[35835]: osdmap e61: 9 total, 9 up, 9 in 2024-04-13T03:29:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:12 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi188 2024-04-13T03:29:13.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:12 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi188 to 5693M 2024-04-13T03:29:13.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:12 smithi188 ceph-mon[36398]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:13.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:12 smithi188 ceph-mon[36398]: osdmap e61: 9 total, 9 up, 9 in 2024-04-13T03:29:13.492 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:29:14.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:14 smithi016 ceph-mon[31772]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:14.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:14 smithi016 ceph-mon[31772]: osdmap e62: 9 total, 9 up, 9 in 2024-04-13T03:29:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:14 smithi119 ceph-mon[35835]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:14 smithi119 ceph-mon[35835]: osdmap e62: 9 total, 9 up, 9 in 2024-04-13T03:29:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:14 smithi188 ceph-mon[36398]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:14 smithi188 ceph-mon[36398]: osdmap e62: 9 total, 9 up, 9 in 2024-04-13T03:29:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.894+0000 7f24fa000700 1 -- 172.21.15.16:0/1145918978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f4075510 msgr2=0x7f24f4075990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.894+0000 7f24fa000700 1 --2- 172.21.15.16:0/1145918978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f4075510 0x7f24f4075990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009990 tx=0x7f24e0009ca0 comp rx=0 tx=0).stop 2024-04-13T03:29:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.894+0000 7f24fa000700 1 -- 172.21.15.16:0/1145918978 shutdown_connections 2024-04-13T03:29:15.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.894+0000 7f24fa000700 1 --2- 172.21.15.16:0/1145918978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f410a9f0 0x7f24f410ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.894+0000 7f24fa000700 1 --2- 172.21.15.16:0/1145918978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24f4076ae0 0x7f24f4074f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.894+0000 7f24fa000700 1 --2- 172.21.15.16:0/1145918978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f4075510 0x7f24f4075990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.894+0000 7f24fa000700 1 -- 172.21.15.16:0/1145918978 >> 172.21.15.16:0/1145918978 conn(0x7f24f40fe080 msgr2=0x7f24f4100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:29:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.895+0000 7f24fa000700 1 -- 172.21.15.16:0/1145918978 shutdown_connections 2024-04-13T03:29:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.895+0000 7f24fa000700 1 -- 172.21.15.16:0/1145918978 wait complete. 2024-04-13T03:29:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.896+0000 7f24fa000700 1 Processor -- start 2024-04-13T03:29:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.896+0000 7f24fa000700 1 -- start start 2024-04-13T03:29:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24fa000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f4075510 0x7f24f419afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:29:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24fa000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24f4076ae0 0x7f24f419b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:29:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24fa000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f410a9f0 0x7f24f41a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:29:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24fa000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24f4110950 con 0x7f24f4075510 2024-04-13T03:29:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24fa000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f24f41107d0 con 0x7f24f410a9f0 2024-04-13T03:29:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24fa000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f24f4110ad0 con 0x7f24f4076ae0 2024-04-13T03:29:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24f3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f4075510 0x7f24f419afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:29:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24f2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24f4076ae0 0x7f24f419b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24f2a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24f4076ae0 0x7f24f419b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:33610/0 (socket says 172.21.15.16:33610) 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24f3e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f410a9f0 0x7f24f41a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24f2a00700 1 -- 172.21.15.16:0/3754864415 learned_addr learned my addr 172.21.15.16:0/3754864415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.897+0000 7f24f3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f4075510 0x7f24f419afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48360/0 (socket says 172.21.15.16:48360) 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.898+0000 7f24f2a00700 1 -- 172.21.15.16:0/3754864415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f410a9f0 msgr2=0x7f24f41a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.898+0000 7f24f2a00700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f410a9f0 0x7f24f41a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.898+0000 7f24f2a00700 1 -- 172.21.15.16:0/3754864415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f4075510 msgr2=0x7f24f419afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.898+0000 7f24f2a00700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f4075510 0x7f24f419afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.898+0000 7f24f2a00700 1 -- 172.21.15.16:0/3754864415 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f41a2c20 con 0x7f24f4076ae0 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.898+0000 7f24f3e00700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f410a9f0 0x7f24f41a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:29:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.898+0000 7f24f2a00700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24f4076ae0 0x7f24f419b500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f24e0000c00 tx=0x7f24e000bbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:29:15.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.903+0000 7f24ebe00700 1 -- 172.21.15.16:0/3754864415 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0026070 con 0x7f24f4076ae0 2024-04-13T03:29:15.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.903+0000 7f24ebe00700 1 -- 172.21.15.16:0/3754864415 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e000f460 con 0x7f24f4076ae0 2024-04-13T03:29:15.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.903+0000 7f24ebe00700 1 -- 172.21.15.16:0/3754864415 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0021670 con 0x7f24f4076ae0 2024-04-13T03:29:15.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.903+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f41a2eb0 con 0x7f24f4076ae0 2024-04-13T03:29:15.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.903+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f41a3250 con 0x7f24f4076ae0 2024-04-13T03:29:15.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.905+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f402dfa0 con 0x7f24f4076ae0 2024-04-13T03:29:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.906+0000 7f24ebe00700 1 -- 172.21.15.16:0/3754864415 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f24e0029070 con 0x7f24f4076ae0 2024-04-13T03:29:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.908+0000 7f24ebe00700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d80a3cc0 0x7f24d80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:29:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.908+0000 7f24ebe00700 1 -- 172.21.15.16:0/3754864415 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f24e00d0900 con 0x7f24f4076ae0 2024-04-13T03:29:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.908+0000 7f24f3400700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d80a3cc0 0x7f24d80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:29:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.909+0000 7f24f3400700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d80a3cc0 0x7f24d80a6180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009680 tx=0x7f24e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:29:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:15.912+0000 7f24ebe00700 1 -- 172.21.15.16:0/3754864415 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24e005dff0 con 0x7f24f4076ae0 2024-04-13T03:29:16.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:15 smithi016 ceph-mon[31772]: osdmap e63: 9 total, 9 up, 9 in 2024-04-13T03:29:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:16.154+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f24f4062ef0 con 0x7f24d80a3cc0 2024-04-13T03:29:16.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:15 smithi119 ceph-mon[35835]: osdmap e63: 9 total, 9 up, 9 in 2024-04-13T03:29:16.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:15 smithi188 ceph-mon[36398]: osdmap e63: 9 total, 9 up, 9 in 2024-04-13T03:29:17.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:16 smithi016 ceph-mon[31772]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:17.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:16 smithi016 ceph-mon[31772]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:29:17.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:16 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:29:17.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:16 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:29:17.177 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:16 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:16 smithi119 ceph-mon[35835]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:16 smithi119 ceph-mon[35835]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:29:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:16 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:29:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:16 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:29:17.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:16 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:16 smithi188 ceph-mon[36398]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:16 smithi188 ceph-mon[36398]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:29:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:16 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:29:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:16 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:29:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:16 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:19.052 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:18 smithi016 ceph-mon[31772]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-13T03:29:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:18 smithi119 ceph-mon[35835]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-13T03:29:19.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:18 smithi188 ceph-mon[36398]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-13T03:29:21.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:20 smithi016 ceph-mon[31772]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-13T03:29:21.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:20 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1246664502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19ad4b60-9404-402d-bd53-bcd5a7887f35"}]: dispatch 2024-04-13T03:29:21.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:20 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1246664502' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19ad4b60-9404-402d-bd53-bcd5a7887f35"}]': finished 2024-04-13T03:29:21.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:20 smithi016 ceph-mon[31772]: osdmap e64: 10 total, 9 up, 10 in 2024-04-13T03:29:21.056 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:21.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:20 smithi119 ceph-mon[35835]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-13T03:29:21.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:20 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1246664502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19ad4b60-9404-402d-bd53-bcd5a7887f35"}]: dispatch 2024-04-13T03:29:21.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:20 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1246664502' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19ad4b60-9404-402d-bd53-bcd5a7887f35"}]': finished 2024-04-13T03:29:21.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:20 smithi119 ceph-mon[35835]: osdmap e64: 10 total, 9 up, 10 in 2024-04-13T03:29:21.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:21.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:20 smithi188 ceph-mon[36398]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-13T03:29:21.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:20 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1246664502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19ad4b60-9404-402d-bd53-bcd5a7887f35"}]: dispatch 2024-04-13T03:29:21.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:20 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1246664502' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19ad4b60-9404-402d-bd53-bcd5a7887f35"}]': finished 2024-04-13T03:29:21.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:20 smithi188 ceph-mon[36398]: osdmap e64: 10 total, 9 up, 10 in 2024-04-13T03:29:21.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:22.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:21 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3738302865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:29:22.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:21 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3738302865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:29:22.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:21 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3738302865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:29:23.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:22 smithi188 ceph-mon[36398]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-13T03:29:23.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:22 smithi119 ceph-mon[35835]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-13T03:29:23.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:22 smithi016 ceph-mon[31772]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-13T03:29:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:24 smithi016 ceph-mon[31772]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-13T03:29:25.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:24 smithi119 ceph-mon[35835]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-13T03:29:25.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:24 smithi188 ceph-mon[36398]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-13T03:29:26.961 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:26 smithi016 ceph-mon[31772]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-13T03:29:27.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:26 smithi119 ceph-mon[35835]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-13T03:29:27.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:26 smithi188 ceph-mon[36398]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-13T03:29:29.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:28 smithi016 ceph-mon[31772]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:29.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:28 smithi119 ceph-mon[35835]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:29.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:28 smithi188 ceph-mon[36398]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:31.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:30 smithi016 ceph-mon[31772]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:31.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-13T03:29:31.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:31.198 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:30 smithi016 ceph-mon[31772]: Deploying daemon osd.9 on smithi016 2024-04-13T03:29:31.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:30 smithi119 ceph-mon[35835]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:31.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-13T03:29:31.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:31.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:30 smithi119 ceph-mon[35835]: Deploying daemon osd.9 on smithi016 2024-04-13T03:29:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:30 smithi188 ceph-mon[36398]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-13T03:29:31.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:31.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:30 smithi188 ceph-mon[36398]: Deploying daemon osd.9 on smithi016 2024-04-13T03:29:33.085 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:32 smithi016 ceph-mon[31772]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:33.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:32 smithi119 ceph-mon[35835]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:32 smithi188 ceph-mon[36398]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:35.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:34 smithi016 ceph-mon[31772]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:35.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:34 smithi119 ceph-mon[35835]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:35.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:34 smithi188 ceph-mon[36398]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:36 smithi119 ceph-mon[35835]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:37.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:36 smithi016 ceph-mon[31772]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:37.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:36 smithi188 ceph-mon[36398]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:38 smithi119 ceph-mon[35835]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:39.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:38 smithi016 ceph-mon[31772]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:39.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:39.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:39.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:39.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:38 smithi188 ceph-mon[36398]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:39.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:39.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:39.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:40 smithi016 ceph-mon[31772]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:41.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:40 smithi119 ceph-mon[35835]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:41.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:40 smithi188 ceph-mon[36398]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:42 smithi016 ceph-mon[31772]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:42 smithi016 ceph-mon[31772]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-13T03:29:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:42 smithi119 ceph-mon[35835]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:42 smithi119 ceph-mon[35835]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-13T03:29:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:43.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:42 smithi188 ceph-mon[36398]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:43.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:42 smithi188 ceph-mon[36398]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-13T03:29:43.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:43.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.052+0000 7f24ebe00700 1 -- 172.21.15.16:0/3754864415 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f24f4062ef0 con 0x7f24d80a3cc0 2024-04-13T03:29:44.055 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 9 on host 'smithi016' 2024-04-13T03:29:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.056+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d80a3cc0 msgr2=0x7f24d80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.056+0000 7f24fa000700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d80a3cc0 0x7f24d80a6180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009680 tx=0x7f24e4009260 comp rx=0 tx=0).stop 2024-04-13T03:29:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.056+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24f4076ae0 msgr2=0x7f24f419b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:44.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.056+0000 7f24fa000700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24f4076ae0 0x7f24f419b500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f24e0000c00 tx=0x7f24e000bbb0 comp rx=0 tx=0).stop 2024-04-13T03:29:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.057+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 shutdown_connections 2024-04-13T03:29:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.057+0000 7f24fa000700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d80a3cc0 0x7f24d80a6180 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.057+0000 7f24fa000700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f410a9f0 0x7f24f41a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.057+0000 7f24fa000700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f4075510 0x7f24f419afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.057+0000 7f24fa000700 1 --2- 172.21.15.16:0/3754864415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24f4076ae0 0x7f24f419b500 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.057+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 >> 172.21.15.16:0/3754864415 conn(0x7f24f40fe080 msgr2=0x7f24f41068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:29:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.057+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 shutdown_connections 2024-04-13T03:29:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:29:44.057+0000 7f24fa000700 1 -- 172.21.15.16:0/3754864415 wait complete. 2024-04-13T03:29:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:43 smithi119 ceph-mon[35835]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-13T03:29:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:43 smithi119 ceph-mon[35835]: osdmap e65: 10 total, 9 up, 10 in 2024-04-13T03:29:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:43 smithi119 ceph-mon[35835]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:29:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:43 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:43 smithi119 ceph-mon[35835]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:29:44.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:43 smithi016 ceph-mon[31772]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-13T03:29:44.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:43 smithi016 ceph-mon[31772]: osdmap e65: 10 total, 9 up, 10 in 2024-04-13T03:29:44.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:43 smithi016 ceph-mon[31772]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:29:44.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:43 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:44.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:43 smithi016 ceph-mon[31772]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:29:44.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:43 smithi188 ceph-mon[36398]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-13T03:29:44.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:43 smithi188 ceph-mon[36398]: osdmap e65: 10 total, 9 up, 10 in 2024-04-13T03:29:44.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:43 smithi188 ceph-mon[36398]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-13T03:29:44.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:43 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:44.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:43 smithi188 ceph-mon[36398]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-13T03:29:44.915 DEBUG:teuthology.orchestra.run.smithi016:osd.9> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.9.service 2024-04-13T03:29:44.920 INFO:tasks.cephadm:Deploying osd.10 on smithi119 with /dev/vg_nvme/lv_1... 2024-04-13T03:29:44.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-13T03:29:45.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:44 smithi016 ceph-mon[31772]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:45.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:44 smithi016 ceph-mon[31772]: osdmap e66: 10 total, 9 up, 10 in 2024-04-13T03:29:45.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:45.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:45.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:45.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:44 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:45.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:44 smithi016 ceph-mon[31772]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' 2024-04-13T03:29:45.170 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:29:45.291 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:44 smithi119 ceph-mon[35835]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:45.291 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:44 smithi119 ceph-mon[35835]: osdmap e66: 10 total, 9 up, 10 in 2024-04-13T03:29:45.291 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:44 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:45.291 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:44 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:45.292 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:44 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:45.292 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:44 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:45.292 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:44 smithi119 ceph-mon[35835]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' 2024-04-13T03:29:45.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:44 smithi188 ceph-mon[36398]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:45.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:44 smithi188 ceph-mon[36398]: osdmap e66: 10 total, 9 up, 10 in 2024-04-13T03:29:45.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:44 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:45.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:44 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:45.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:44 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:45.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:44 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:45.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:44 smithi188 ceph-mon[36398]: from='osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905]' entity='osd.9' 2024-04-13T03:29:46.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:45 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:29:46.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:45 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:29:46.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:45 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:46.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:45 smithi016 ceph-mon[31772]: osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] boot 2024-04-13T03:29:46.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:45 smithi016 ceph-mon[31772]: osdmap e67: 10 total, 10 up, 10 in 2024-04-13T03:29:46.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:45 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:46.312 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:45 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:29:46.312 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:45 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:29:46.312 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:45 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:46.312 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:45 smithi119 ceph-mon[35835]: osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] boot 2024-04-13T03:29:46.312 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:45 smithi119 ceph-mon[35835]: osdmap e67: 10 total, 10 up, 10 in 2024-04-13T03:29:46.312 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:45 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:46.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:45 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:29:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:45 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:29:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:45 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:45 smithi188 ceph-mon[36398]: osd.9 [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] boot 2024-04-13T03:29:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:45 smithi188 ceph-mon[36398]: osdmap e67: 10 total, 10 up, 10 in 2024-04-13T03:29:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:45 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:29:47.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:46 smithi016 ceph-mon[31772]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:47.194 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:46 smithi016 ceph-mon[31772]: osdmap e68: 10 total, 10 up, 10 in 2024-04-13T03:29:47.258 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-13T03:29:47.258 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-13T03:29:47.258 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-04-13T03:29:47.258 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-04-13T03:29:47.258 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112489 s, 932 MB/s 2024-04-13T03:29:47.258 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-04-13T03:29:47.259 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:46 smithi119 ceph-mon[35835]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:47.259 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:46 smithi119 ceph-mon[35835]: osdmap e68: 10 total, 10 up, 10 in 2024-04-13T03:29:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:46 smithi188 ceph-mon[36398]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-13T03:29:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:46 smithi188 ceph-mon[36398]: osdmap e68: 10 total, 10 up, 10 in 2024-04-13T03:29:47.877 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_1 2024-04-13T03:29:48.127 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:29:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:48 smithi119 ceph-mon[35835]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:48 smithi016 ceph-mon[31772]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:48 smithi188 ceph-mon[36398]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:49.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.858+0000 7f22e01bf700 1 -- 172.21.15.119:0/2292424673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d8102c10 msgr2=0x7f22d8103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:49.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.858+0000 7f22e01bf700 1 --2- 172.21.15.119:0/2292424673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d8102c10 0x7f22d8103070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f22d40099c0 tx=0x7f22d4009cd0 comp rx=0 tx=0).stop 2024-04-13T03:29:49.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.858+0000 7f22e01bf700 1 -- 172.21.15.119:0/2292424673 shutdown_connections 2024-04-13T03:29:49.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.858+0000 7f22e01bf700 1 --2- 172.21.15.119:0/2292424673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d81035b0 0x7f22d8109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:49.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.858+0000 7f22e01bf700 1 --2- 172.21.15.119:0/2292424673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d8101a10 0x7f22d8101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:49.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.858+0000 7f22e01bf700 1 --2- 172.21.15.119:0/2292424673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d8102c10 0x7f22d8103070 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:49.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.858+0000 7f22e01bf700 1 -- 172.21.15.119:0/2292424673 >> 172.21.15.119:0/2292424673 conn(0x7f22d80fcfb0 msgr2=0x7f22d80ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:29:49.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.858+0000 7f22e01bf700 1 -- 172.21.15.119:0/2292424673 shutdown_connections 2024-04-13T03:29:49.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.858+0000 7f22e01bf700 1 -- 172.21.15.119:0/2292424673 wait complete. 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22e01bf700 1 Processor -- start 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22e01bf700 1 -- start start 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22e01bf700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d8101a10 0x7f22d8198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22e01bf700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8102c10 0x7f22d81993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22e01bf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d81035b0 0x7f22d81a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22e01bf700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22d810d6a0 con 0x7f22d8102c10 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22e01bf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f22d810d520 con 0x7f22d81035b0 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22e01bf700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f22d810d820 con 0x7f22d8101a10 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22de75c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d81035b0 0x7f22d81a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22ddf5b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d8101a10 0x7f22d8198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22de75c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d81035b0 0x7f22d81a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51378/0 (socket says 172.21.15.119:51378) 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22de75c700 1 -- 172.21.15.119:0/3030966324 learned_addr learned my addr 172.21.15.119:0/3030966324 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22dd75a700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8102c10 0x7f22d81993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:29:49.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22de75c700 1 -- 172.21.15.119:0/3030966324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d8101a10 msgr2=0x7f22d8198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22de75c700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d8101a10 0x7f22d8198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22de75c700 1 -- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8102c10 msgr2=0x7f22d81993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22de75c700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8102c10 0x7f22d81993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.859+0000 7f22de75c700 1 -- 172.21.15.119:0/3030966324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d81a0a80 con 0x7f22d81035b0 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.860+0000 7f22de75c700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d81035b0 0x7f22d81a03e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f22d000b970 tx=0x7f22d000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.860+0000 7f22ceffd700 1 -- 172.21.15.119:0/3030966324 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0019070 con 0x7f22d81035b0 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.860+0000 7f22ceffd700 1 -- 172.21.15.119:0/3030966324 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22d000ce10 con 0x7f22d81035b0 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.860+0000 7f22ceffd700 1 -- 172.21.15.119:0/3030966324 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d00073d0 con 0x7f22d81035b0 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.860+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d81a0d70 con 0x7f22d81035b0 2024-04-13T03:29:49.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.860+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d81a1280 con 0x7f22d81035b0 2024-04-13T03:29:49.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.861+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22d80697d0 con 0x7f22d81035b0 2024-04-13T03:29:49.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.862+0000 7f22ceffd700 1 -- 172.21.15.119:0/3030966324 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f22d001c070 con 0x7f22d81035b0 2024-04-13T03:29:49.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.862+0000 7f22ceffd700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f22c40a3cc0 0x7f22c40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:29:49.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.863+0000 7f22ceffd700 1 -- 172.21.15.119:0/3030966324 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f22d00cf3d0 con 0x7f22d81035b0 2024-04-13T03:29:49.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.863+0000 7f22ddf5b700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f22c40a3cc0 0x7f22c40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:29:49.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.864+0000 7f22ddf5b700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f22c40a3cc0 0x7f22c40a6180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f22d8102a70 tx=0x7f22c800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:29:49.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:49.866+0000 7f22ceffd700 1 -- 172.21.15.119:0/3030966324 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22d80697d0 con 0x7f22d81035b0 2024-04-13T03:29:50.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:29:50.071+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f22d80ff1a0 con 0x7f22c40a3cc0 2024-04-13T03:29:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:29:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:29:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:29:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi016 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi016 to 4273M 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:51.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:51.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:51.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi016 2024-04-13T03:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi016 to 4273M 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:51.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:29:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi016 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi016 to 4273M 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:51.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:51.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:51.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:51.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:29:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:53 smithi119 ceph-mon[35835]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:53 smithi119 ceph-mon[35835]: from='client.? 172.21.15.119:0/1429014008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c665a9ff-a65b-4ee2-936d-4169ea0ac9fa"}]: dispatch 2024-04-13T03:29:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:53 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c665a9ff-a65b-4ee2-936d-4169ea0ac9fa"}]: dispatch 2024-04-13T03:29:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:53 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c665a9ff-a65b-4ee2-936d-4169ea0ac9fa"}]': finished 2024-04-13T03:29:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:53 smithi119 ceph-mon[35835]: osdmap e69: 11 total, 10 up, 11 in 2024-04-13T03:29:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:53 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:29:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:53 smithi016 ceph-mon[31772]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:53.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:53 smithi016 ceph-mon[31772]: from='client.? 172.21.15.119:0/1429014008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c665a9ff-a65b-4ee2-936d-4169ea0ac9fa"}]: dispatch 2024-04-13T03:29:53.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:53 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c665a9ff-a65b-4ee2-936d-4169ea0ac9fa"}]: dispatch 2024-04-13T03:29:53.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:53 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c665a9ff-a65b-4ee2-936d-4169ea0ac9fa"}]': finished 2024-04-13T03:29:53.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:53 smithi016 ceph-mon[31772]: osdmap e69: 11 total, 10 up, 11 in 2024-04-13T03:29:53.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:53 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:29:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:53 smithi188 ceph-mon[36398]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:53 smithi188 ceph-mon[36398]: from='client.? 172.21.15.119:0/1429014008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c665a9ff-a65b-4ee2-936d-4169ea0ac9fa"}]: dispatch 2024-04-13T03:29:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:53 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c665a9ff-a65b-4ee2-936d-4169ea0ac9fa"}]: dispatch 2024-04-13T03:29:53.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:53 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c665a9ff-a65b-4ee2-936d-4169ea0ac9fa"}]': finished 2024-04-13T03:29:53.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:53 smithi188 ceph-mon[36398]: osdmap e69: 11 total, 10 up, 11 in 2024-04-13T03:29:53.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:53 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:29:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:54 smithi119 ceph-mon[35835]: from='client.? 172.21.15.119:0/3725965074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:29:54.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:54 smithi016 ceph-mon[31772]: from='client.? 172.21.15.119:0/3725965074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:29:54.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:54 smithi188 ceph-mon[36398]: from='client.? 172.21.15.119:0/3725965074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:29:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:55 smithi119 ceph-mon[35835]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:55.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:55 smithi016 ceph-mon[31772]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:55.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:55 smithi188 ceph-mon[36398]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:57 smithi119 ceph-mon[35835]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:57 smithi016 ceph-mon[31772]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:57 smithi188 ceph-mon[36398]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:59.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:29:59 smithi119 ceph-mon[35835]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:29:59 smithi016 ceph-mon[31772]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:29:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:29:59 smithi188 ceph-mon[36398]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:00.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:00 smithi119 ceph-mon[35835]: overall HEALTH_OK 2024-04-13T03:30:00.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:00 smithi016 ceph-mon[31772]: overall HEALTH_OK 2024-04-13T03:30:00.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:00 smithi188 ceph-mon[36398]: overall HEALTH_OK 2024-04-13T03:30:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:01 smithi119 ceph-mon[35835]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-13T03:30:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:01 smithi016 ceph-mon[31772]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-13T03:30:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:01 smithi188 ceph-mon[36398]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:01.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-13T03:30:01.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:02 smithi119 ceph-mon[35835]: Deploying daemon osd.10 on smithi119 2024-04-13T03:30:02.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:02 smithi016 ceph-mon[31772]: Deploying daemon osd.10 on smithi119 2024-04-13T03:30:02.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:02 smithi188 ceph-mon[36398]: Deploying daemon osd.10 on smithi119 2024-04-13T03:30:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:03 smithi119 ceph-mon[35835]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:03 smithi016 ceph-mon[31772]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:03 smithi188 ceph-mon[36398]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:05 smithi119 ceph-mon[35835]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:05 smithi016 ceph-mon[31772]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:05 smithi188 ceph-mon[36398]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:07 smithi119 ceph-mon[35835]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:07 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:07 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:07 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:07 smithi016 ceph-mon[31772]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:07.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:07.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:07 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:07 smithi188 ceph-mon[36398]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:07.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:07 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:07.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:07 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:07.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:07 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:09.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:09 smithi119 ceph-mon[35835]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:09.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:09 smithi016 ceph-mon[31772]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:09 smithi188 ceph-mon[36398]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:10.513 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 10 on host 'smithi119' 2024-04-13T03:30:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.513+0000 7f22ceffd700 1 -- 172.21.15.119:0/3030966324 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f22d80ff1a0 con 0x7f22c40a3cc0 2024-04-13T03:30:10.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f22c40a3cc0 msgr2=0x7f22c40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f22c40a3cc0 0x7f22c40a6180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f22d8102a70 tx=0x7f22c800b410 comp rx=0 tx=0).stop 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d81035b0 msgr2=0x7f22d81a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d81035b0 0x7f22d81a03e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f22d000b970 tx=0x7f22d000bc80 comp rx=0 tx=0).stop 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 shutdown_connections 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f22c40a3cc0 0x7f22c40a6180 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22d81035b0 0x7f22d81a03e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22d8101a10 0x7f22d8198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 --2- 172.21.15.119:0/3030966324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22d8102c10 0x7f22d81993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 >> 172.21.15.119:0/3030966324 conn(0x7f22d80fcfb0 msgr2=0x7f22d80fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 shutdown_connections 2024-04-13T03:30:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-13T03:30:10.515+0000 7f22e01bf700 1 -- 172.21.15.119:0/3030966324 wait complete. 2024-04-13T03:30:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:11.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:11.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:11.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='osd.10 [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-13T03:30:11.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:10 smithi119 ceph-mon[35835]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='osd.10 [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-13T03:30:11.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:10 smithi016 ceph-mon[31772]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-13T03:30:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='osd.10 [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-13T03:30:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:10 smithi188 ceph-mon[36398]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-13T03:30:11.245 DEBUG:teuthology.orchestra.run.smithi119:osd.10> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.10.service 2024-04-13T03:30:11.248 INFO:tasks.cephadm:Deploying osd.11 on smithi188 with /dev/vg_nvme/lv_1... 2024-04-13T03:30:11.248 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-13T03:30:11.506 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:30:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:12 smithi119 ceph-mon[35835]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-13T03:30:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:12 smithi119 ceph-mon[35835]: from='osd.10 [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:30:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:12 smithi119 ceph-mon[35835]: osdmap e70: 11 total, 10 up, 11 in 2024-04-13T03:30:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:12 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:12 smithi119 ceph-mon[35835]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:30:12.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:12 smithi188 ceph-mon[36398]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-13T03:30:12.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:12 smithi188 ceph-mon[36398]: from='osd.10 [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:30:12.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:12 smithi188 ceph-mon[36398]: osdmap e70: 11 total, 10 up, 11 in 2024-04-13T03:30:12.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:12 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:12.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:12 smithi188 ceph-mon[36398]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:30:12.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:12 smithi016 ceph-mon[31772]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-13T03:30:12.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:12 smithi016 ceph-mon[31772]: from='osd.10 [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:30:12.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:12 smithi016 ceph-mon[31772]: osdmap e70: 11 total, 10 up, 11 in 2024-04-13T03:30:12.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:12 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:12.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:12 smithi016 ceph-mon[31772]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-13T03:30:13.322 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:30:12 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:30:12.952+0000 7f477aa90700 -1 osd.10 0 waiting for initial osdmap 2024-04-13T03:30:13.322 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:30:12 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:30:12.969+0000 7f477207a700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T03:30:13.544 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-13T03:30:13.544 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-13T03:30:13.545 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-13T03:30:13.545 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-13T03:30:13.545 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131123 s, 800 MB/s 2024-04-13T03:30:13.545 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-13T03:30:13.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:13 smithi119 ceph-mon[35835]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:13.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:13 smithi119 ceph-mon[35835]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:30:13.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:13 smithi119 ceph-mon[35835]: osdmap e71: 11 total, 10 up, 11 in 2024-04-13T03:30:13.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:13 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:13.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:13 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:13.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:13 smithi119 ceph-mon[35835]: osdmap e72: 11 total, 10 up, 11 in 2024-04-13T03:30:13.680 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:13 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:13.680 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:13 smithi119 ceph-mon[35835]: from='osd.10 ' entity='osd.10' 2024-04-13T03:30:13.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:13 smithi016 ceph-mon[31772]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:13.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:13 smithi016 ceph-mon[31772]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:30:13.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:13 smithi016 ceph-mon[31772]: osdmap e71: 11 total, 10 up, 11 in 2024-04-13T03:30:13.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:13 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:13.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:13 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:13.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:13 smithi016 ceph-mon[31772]: osdmap e72: 11 total, 10 up, 11 in 2024-04-13T03:30:13.786 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:13 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:13.786 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:13 smithi016 ceph-mon[31772]: from='osd.10 ' entity='osd.10' 2024-04-13T03:30:13.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:13 smithi188 ceph-mon[36398]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:13.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:13 smithi188 ceph-mon[36398]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-13T03:30:13.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:13 smithi188 ceph-mon[36398]: osdmap e71: 11 total, 10 up, 11 in 2024-04-13T03:30:13.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:13 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:13.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:13 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:13.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:13 smithi188 ceph-mon[36398]: osdmap e72: 11 total, 10 up, 11 in 2024-04-13T03:30:13.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:13 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:13.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:13 smithi188 ceph-mon[36398]: from='osd.10 ' entity='osd.10' 2024-04-13T03:30:14.201 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_1 2024-04-13T03:30:14.443 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:30:14.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:30:14.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:30:14.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:14.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: osd.10 [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] boot 2024-04-13T03:30:14.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: osdmap e73: 11 total, 11 up, 11 in 2024-04-13T03:30:14.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:14.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:14.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:14.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:30:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:30:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: osd.10 [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] boot 2024-04-13T03:30:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: osdmap e73: 11 total, 11 up, 11 in 2024-04-13T03:30:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:14.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:14.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: osd.10 [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] boot 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: osdmap e73: 11 total, 11 up, 11 in 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:14.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:14.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:14.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:14.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:15.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:15 smithi188 ceph-mon[36398]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:15.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:15 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi119 2024-04-13T03:30:15.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:15 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi119 to 4270M 2024-04-13T03:30:15.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:15 smithi188 ceph-mon[36398]: osdmap e74: 11 total, 11 up, 11 in 2024-04-13T03:30:15.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:15 smithi119 ceph-mon[35835]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:15.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:15 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi119 2024-04-13T03:30:15.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:15 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi119 to 4270M 2024-04-13T03:30:15.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:15 smithi119 ceph-mon[35835]: osdmap e74: 11 total, 11 up, 11 in 2024-04-13T03:30:15.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:15 smithi016 ceph-mon[31772]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-04-13T03:30:15.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:15 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi119 2024-04-13T03:30:15.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:15 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi119 to 4270M 2024-04-13T03:30:15.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:15 smithi016 ceph-mon[31772]: osdmap e74: 11 total, 11 up, 11 in 2024-04-13T03:30:16.132 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.131+0000 7fb3515de700 1 -- 172.21.15.188:0/67726258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c105f30 msgr2=0x7fb34c108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:16.132 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.131+0000 7fb3515de700 1 --2- 172.21.15.188:0/67726258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c105f30 0x7fb34c108320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009990 tx=0x7fb33c009ca0 comp rx=0 tx=0).stop 2024-04-13T03:30:16.132 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.131+0000 7fb3515de700 1 -- 172.21.15.188:0/67726258 shutdown_connections 2024-04-13T03:30:16.132 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.131+0000 7fb3515de700 1 --2- 172.21.15.188:0/67726258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb34c108860 0x7fb34c10ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:16.132 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.131+0000 7fb3515de700 1 --2- 172.21.15.188:0/67726258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb34c103600 0x7fb34c1059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:16.132 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.131+0000 7fb3515de700 1 --2- 172.21.15.188:0/67726258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c105f30 0x7fb34c108320 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:16.132 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.131+0000 7fb3515de700 1 -- 172.21.15.188:0/67726258 >> 172.21.15.188:0/67726258 conn(0x7fb34c0fcfd0 msgr2=0x7fb34c0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:16.132 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.131+0000 7fb3515de700 1 -- 172.21.15.188:0/67726258 shutdown_connections 2024-04-13T03:30:16.132 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.131+0000 7fb3515de700 1 -- 172.21.15.188:0/67726258 wait complete. 2024-04-13T03:30:16.134 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb3515de700 1 Processor -- start 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb3515de700 1 -- start start 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb3515de700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb34c103600 0x7fb34c102f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb3515de700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb34c105f30 0x7fb34c1015b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb3515de700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c108860 0x7fb34c101af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb3515de700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c10e7e0 con 0x7fb34c105f30 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb3515de700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c10e660 con 0x7fb34c103600 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb3515de700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c10e960 con 0x7fb34c108860 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb34b7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c108860 0x7fb34c101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb34b7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c108860 0x7fb34c101af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:39252/0 (socket says 172.21.15.188:39252) 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb34b7fe700 1 -- 172.21.15.188:0/3111595872 learned_addr learned my addr 172.21.15.188:0/3111595872 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb34affd700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb34c103600 0x7fb34c102f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb34b7fe700 1 -- 172.21.15.188:0/3111595872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb34c103600 msgr2=0x7fb34c102f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:16.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb34b7fe700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb34c103600 0x7fb34c102f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb34b7fe700 1 -- 172.21.15.188:0/3111595872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb34c105f30 msgr2=0x7fb34c1015b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb34b7fe700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb34c105f30 0x7fb34c1015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.132+0000 7fb34b7fe700 1 -- 172.21.15.188:0/3111595872 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34c1023a0 con 0x7fb34c108860 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.133+0000 7fb34b7fe700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c108860 0x7fb34c101af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb34400b9e0 tx=0x7fb34400bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.133+0000 7fb333fff700 1 -- 172.21.15.188:0/3111595872 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb344019070 con 0x7fb34c108860 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.133+0000 7fb333fff700 1 -- 172.21.15.188:0/3111595872 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb344014440 con 0x7fb34c108860 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.133+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb34c1a2fd0 con 0x7fb34c108860 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.133+0000 7fb333fff700 1 -- 172.21.15.188:0/3111595872 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3440125a0 con 0x7fb34c108860 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.133+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb34c1a34b0 con 0x7fb34c108860 2024-04-13T03:30:16.136 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.134+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb34c0697a0 con 0x7fb34c108860 2024-04-13T03:30:16.137 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.135+0000 7fb333fff700 1 -- 172.21.15.188:0/3111595872 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb34401c070 con 0x7fb34c108860 2024-04-13T03:30:16.137 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.135+0000 7fb333fff700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb3340a3c40 0x7fb3340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:16.137 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.136+0000 7fb333fff700 1 -- 172.21.15.188:0/3111595872 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6723+0+0 (secure 0 0 0) 0x7fb3440cf180 con 0x7fb34c108860 2024-04-13T03:30:16.137 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.136+0000 7fb34affd700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb3340a3c40 0x7fb3340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:16.137 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.137+0000 7fb34affd700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb3340a3c40 0x7fb3340a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb340006fd0 tx=0x7fb340008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:16.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.141+0000 7fb333fff700 1 -- 172.21.15.188:0/3111595872 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb34409c190 con 0x7fb34c108860 2024-04-13T03:30:16.349 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:16.348+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb34c062ef0 con 0x7fb3340a3c40 2024-04-13T03:30:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:16 smithi119 ceph-mon[35835]: osdmap e75: 11 total, 11 up, 11 in 2024-04-13T03:30:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:16 smithi119 ceph-mon[35835]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:16 smithi119 ceph-mon[35835]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:30:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:16 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:30:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:16 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:30:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:16 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:17.102 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:16 smithi188 ceph-mon[36398]: osdmap e75: 11 total, 11 up, 11 in 2024-04-13T03:30:17.102 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:16 smithi188 ceph-mon[36398]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:17.102 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:16 smithi188 ceph-mon[36398]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:30:17.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:16 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:30:17.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:16 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:30:17.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:16 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:17.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:16 smithi016 ceph-mon[31772]: osdmap e75: 11 total, 11 up, 11 in 2024-04-13T03:30:17.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:16 smithi016 ceph-mon[31772]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:17.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:16 smithi016 ceph-mon[31772]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:30:17.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:16 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-13T03:30:17.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:16 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-13T03:30:17.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:16 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:18.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:17 smithi119 ceph-mon[35835]: osdmap e76: 11 total, 11 up, 11 in 2024-04-13T03:30:18.099 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:17 smithi188 ceph-mon[36398]: osdmap e76: 11 total, 11 up, 11 in 2024-04-13T03:30:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:17 smithi016 ceph-mon[31772]: osdmap e76: 11 total, 11 up, 11 in 2024-04-13T03:30:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:18 smithi119 ceph-mon[35835]: osdmap e77: 11 total, 11 up, 11 in 2024-04-13T03:30:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:18 smithi119 ceph-mon[35835]: pgmap v266: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-13T03:30:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:18 smithi016 ceph-mon[31772]: osdmap e77: 11 total, 11 up, 11 in 2024-04-13T03:30:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:18 smithi016 ceph-mon[31772]: pgmap v266: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-13T03:30:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:18 smithi188 ceph-mon[36398]: osdmap e77: 11 total, 11 up, 11 in 2024-04-13T03:30:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:18 smithi188 ceph-mon[36398]: pgmap v266: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-13T03:30:20.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:19 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/2922783518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "235b52a6-7965-4637-a723-c5802733e041"}]: dispatch 2024-04-13T03:30:20.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:19 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "235b52a6-7965-4637-a723-c5802733e041"}]: dispatch 2024-04-13T03:30:20.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:19 smithi119 ceph-mon[35835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "235b52a6-7965-4637-a723-c5802733e041"}]': finished 2024-04-13T03:30:20.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:19 smithi119 ceph-mon[35835]: osdmap e78: 12 total, 11 up, 12 in 2024-04-13T03:30:20.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:19 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:20.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:19 smithi119 ceph-mon[35835]: from='client.? 172.21.15.188:0/821699740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:30:20.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:19 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/2922783518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "235b52a6-7965-4637-a723-c5802733e041"}]: dispatch 2024-04-13T03:30:20.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:19 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "235b52a6-7965-4637-a723-c5802733e041"}]: dispatch 2024-04-13T03:30:20.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:19 smithi016 ceph-mon[31772]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "235b52a6-7965-4637-a723-c5802733e041"}]': finished 2024-04-13T03:30:20.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:19 smithi016 ceph-mon[31772]: osdmap e78: 12 total, 11 up, 12 in 2024-04-13T03:30:20.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:19 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:20.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:19 smithi016 ceph-mon[31772]: from='client.? 172.21.15.188:0/821699740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:30:20.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:19 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/2922783518' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "235b52a6-7965-4637-a723-c5802733e041"}]: dispatch 2024-04-13T03:30:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:19 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "235b52a6-7965-4637-a723-c5802733e041"}]: dispatch 2024-04-13T03:30:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:19 smithi188 ceph-mon[36398]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "235b52a6-7965-4637-a723-c5802733e041"}]': finished 2024-04-13T03:30:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:19 smithi188 ceph-mon[36398]: osdmap e78: 12 total, 11 up, 12 in 2024-04-13T03:30:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:19 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:20.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:19 smithi188 ceph-mon[36398]: from='client.? 172.21.15.188:0/821699740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-13T03:30:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:20 smithi119 ceph-mon[35835]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-04-13T03:30:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:20 smithi016 ceph-mon[31772]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-04-13T03:30:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:20 smithi188 ceph-mon[36398]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-04-13T03:30:23.044 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:22 smithi188 ceph-mon[36398]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-13T03:30:23.225 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:22 smithi016 ceph-mon[31772]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-13T03:30:23.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:22 smithi119 ceph-mon[35835]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-13T03:30:24.994 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:24 smithi188 ceph-mon[36398]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-13T03:30:25.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:24 smithi119 ceph-mon[35835]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-13T03:30:25.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:24 smithi016 ceph-mon[31772]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-13T03:30:27.159 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:26 smithi188 ceph-mon[36398]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:27.159 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-13T03:30:27.159 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:27.159 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:26 smithi188 ceph-mon[36398]: Deploying daemon osd.11 on smithi188 2024-04-13T03:30:27.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:26 smithi119 ceph-mon[35835]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:27.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-13T03:30:27.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:27.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:26 smithi119 ceph-mon[35835]: Deploying daemon osd.11 on smithi188 2024-04-13T03:30:27.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:26 smithi016 ceph-mon[31772]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:27.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-13T03:30:27.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:27.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:26 smithi016 ceph-mon[31772]: Deploying daemon osd.11 on smithi188 2024-04-13T03:30:29.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:28 smithi119 ceph-mon[35835]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:29.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:28 smithi016 ceph-mon[31772]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:29.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:28 smithi188 ceph-mon[36398]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:31.277 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:30 smithi188 ceph-mon[36398]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:31.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:30 smithi119 ceph-mon[35835]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:30 smithi016 ceph-mon[31772]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:33 smithi188 ceph-mon[36398]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:33.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:33.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:33 smithi119 ceph-mon[35835]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:33.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:33 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:33.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:33 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:33.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:33 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:33.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:33 smithi016 ceph-mon[31772]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:33.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:33 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:33.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:33 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:33.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:33 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:35.328 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:35 smithi188 ceph-mon[36398]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:35.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:35 smithi119 ceph-mon[35835]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:35.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:35 smithi016 ceph-mon[31772]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:35.682 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 11 on host 'smithi188' 2024-04-13T03:30:35.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.681+0000 7fb333fff700 1 -- 172.21.15.188:0/3111595872 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb34c062ef0 con 0x7fb3340a3c40 2024-04-13T03:30:35.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.683+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb3340a3c40 msgr2=0x7fb3340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:35.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.683+0000 7fb3515de700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb3340a3c40 0x7fb3340a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb340006fd0 tx=0x7fb340008040 comp rx=0 tx=0).stop 2024-04-13T03:30:35.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.683+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c108860 msgr2=0x7fb34c101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:35.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.683+0000 7fb3515de700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c108860 0x7fb34c101af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb34400b9e0 tx=0x7fb34400bda0 comp rx=0 tx=0).stop 2024-04-13T03:30:35.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.684+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 shutdown_connections 2024-04-13T03:30:35.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.684+0000 7fb3515de700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb3340a3c40 0x7fb3340a6100 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:35.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.684+0000 7fb3515de700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb34c108860 0x7fb34c101af0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:35.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.684+0000 7fb3515de700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb34c103600 0x7fb34c102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:35.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.684+0000 7fb3515de700 1 --2- 172.21.15.188:0/3111595872 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb34c105f30 0x7fb34c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:35.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.684+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 >> 172.21.15.188:0/3111595872 conn(0x7fb34c0fcfd0 msgr2=0x7fb34c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:35.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.684+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 shutdown_connections 2024-04-13T03:30:35.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:35.684+0000 7fb3515de700 1 -- 172.21.15.188:0/3111595872 wait complete. 2024-04-13T03:30:36.300 DEBUG:teuthology.orchestra.run.smithi188:osd.11> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.11.service 2024-04-13T03:30:36.302 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-13T03:30:36.303 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd stat -f json 2024-04-13T03:30:36.796 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:30:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:36.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:36 smithi016 ceph-mon[31772]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:36.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:36 smithi016 ceph-mon[31772]: from='osd.11 [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-13T03:30:36.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:36 smithi016 ceph-mon[31772]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-13T03:30:36.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:36.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:36 smithi119 ceph-mon[35835]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:36 smithi119 ceph-mon[35835]: from='osd.11 [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-13T03:30:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:36 smithi119 ceph-mon[35835]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-13T03:30:37.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:37.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:37.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:37.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:37.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:36 smithi188 ceph-mon[36398]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:37.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:36 smithi188 ceph-mon[36398]: from='osd.11 [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-13T03:30:37.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:36 smithi188 ceph-mon[36398]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-13T03:30:37.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:37.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:37 smithi188 ceph-mon[36398]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-13T03:30:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:37 smithi188 ceph-mon[36398]: osdmap e79: 12 total, 11 up, 12 in 2024-04-13T03:30:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:37 smithi188 ceph-mon[36398]: from='osd.11 [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:30:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:37 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:37 smithi188 ceph-mon[36398]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:30:37.873 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:30:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:30:37.651+0000 7f5891c5c700 -1 osd.11 0 waiting for initial osdmap 2024-04-13T03:30:37.873 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:30:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:30:37.663+0000 7f588da4f700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-13T03:30:37.996 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:37 smithi016 ceph-mon[31772]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-13T03:30:37.996 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:37 smithi016 ceph-mon[31772]: osdmap e79: 12 total, 11 up, 12 in 2024-04-13T03:30:37.996 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:37 smithi016 ceph-mon[31772]: from='osd.11 [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:30:37.996 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:37 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:37.996 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:37 smithi016 ceph-mon[31772]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:30:38.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:37 smithi119 ceph-mon[35835]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-13T03:30:38.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:37 smithi119 ceph-mon[35835]: osdmap e79: 12 total, 11 up, 12 in 2024-04-13T03:30:38.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:37 smithi119 ceph-mon[35835]: from='osd.11 [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:30:38.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:37 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:38.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:37 smithi119 ceph-mon[35835]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-13T03:30:38.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:38 smithi188 ceph-mon[36398]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:30:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:38 smithi188 ceph-mon[36398]: osdmap e80: 12 total, 11 up, 12 in 2024-04-13T03:30:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:38 smithi188 ceph-mon[36398]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:38.898 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:38 smithi016 ceph-mon[31772]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:30:38.898 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:38 smithi016 ceph-mon[31772]: osdmap e80: 12 total, 11 up, 12 in 2024-04-13T03:30:38.898 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:38.898 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:38 smithi016 ceph-mon[31772]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:38.898 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:38 smithi119 ceph-mon[35835]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-13T03:30:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:38 smithi119 ceph-mon[35835]: osdmap e80: 12 total, 11 up, 12 in 2024-04-13T03:30:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:38 smithi119 ceph-mon[35835]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-13T03:30:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:39.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.175+0000 7f0311a00700 1 -- 172.21.15.16:0/3942814815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c10a9e0 msgr2=0x7f030c10ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.175+0000 7f0311a00700 1 --2- 172.21.15.16:0/3942814815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c10a9e0 0x7f030c10ce70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f030400b0a0 tx=0x7f030400b3b0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.175+0000 7f0311a00700 1 -- 172.21.15.16:0/3942814815 shutdown_connections 2024-04-13T03:30:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.175+0000 7f0311a00700 1 --2- 172.21.15.16:0/3942814815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c10a9e0 0x7f030c10ce70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.175+0000 7f0311a00700 1 --2- 172.21.15.16:0/3942814815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f030c076ae0 0x7f030c074f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.175+0000 7f0311a00700 1 --2- 172.21.15.16:0/3942814815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f030c075510 0x7f030c075990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.175+0000 7f0311a00700 1 -- 172.21.15.16:0/3942814815 >> 172.21.15.16:0/3942814815 conn(0x7f030c0fe0b0 msgr2=0x7f030c1004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.175+0000 7f0311a00700 1 -- 172.21.15.16:0/3942814815 shutdown_connections 2024-04-13T03:30:39.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.175+0000 7f0311a00700 1 -- 172.21.15.16:0/3942814815 wait complete. 2024-04-13T03:30:39.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.176+0000 7f0311a00700 1 Processor -- start 2024-04-13T03:30:39.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f0311a00700 1 -- start start 2024-04-13T03:30:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f0311a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f030c075510 0x7f030c19af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f0311a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c076ae0 0x7f030c19b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f0311a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f030c10a9e0 0x7f030c1a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f0311a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f030c110940 con 0x7f030c076ae0 2024-04-13T03:30:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f0311a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f030c1107c0 con 0x7f030c10a9e0 2024-04-13T03:30:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f0311a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f030c110ac0 con 0x7f030c075510 2024-04-13T03:30:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f030aa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f030c075510 0x7f030c19af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f030aa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f030c075510 0x7f030c19af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:40522/0 (socket says 172.21.15.16:40522) 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f030aa00700 1 -- 172.21.15.16:0/3328145667 learned_addr learned my addr 172.21.15.16:0/3328145667 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f030b400700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f030c10a9e0 0x7f030c1a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.177+0000 7f030a000700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c076ae0 0x7f030c19b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.178+0000 7f030aa00700 1 -- 172.21.15.16:0/3328145667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f030c10a9e0 msgr2=0x7f030c1a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.178+0000 7f030aa00700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f030c10a9e0 0x7f030c1a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.178+0000 7f030aa00700 1 -- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c076ae0 msgr2=0x7f030c19b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.178+0000 7f030aa00700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c076ae0 0x7f030c19b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.178+0000 7f030aa00700 1 -- 172.21.15.16:0/3328145667 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030c1a2ba0 con 0x7f030c075510 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.178+0000 7f030b400700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f030c10a9e0 0x7f030c1a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:30:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.178+0000 7f030aa00700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f030c075510 0x7f030c19af40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00e9a0 tx=0x7f02fc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.178+0000 7f0303400700 1 -- 172.21.15.16:0/3328145667 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc004a30 con 0x7f030c075510 2024-04-13T03:30:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.179+0000 7f0303400700 1 -- 172.21.15.16:0/3328145667 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02fc00ccf0 con 0x7f030c075510 2024-04-13T03:30:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.179+0000 7f0303400700 1 -- 172.21.15.16:0/3328145667 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc005680 con 0x7f030c075510 2024-04-13T03:30:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.179+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f030c1a2e90 con 0x7f030c075510 2024-04-13T03:30:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.179+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f030c1a32f0 con 0x7f030c075510 2024-04-13T03:30:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.180+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f030c0697a0 con 0x7f030c075510 2024-04-13T03:30:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.182+0000 7f0303400700 1 -- 172.21.15.16:0/3328145667 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f02fc00c810 con 0x7f030c075510 2024-04-13T03:30:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.183+0000 7f0303400700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f02f00a3cc0 0x7f02f00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.183+0000 7f0303400700 1 -- 172.21.15.16:0/3328145667 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f02fc0d00d0 con 0x7f030c075510 2024-04-13T03:30:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.183+0000 7f030a000700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f02f00a3cc0 0x7f02f00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.184+0000 7f030a000700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f02f00a3cc0 0x7f02f00a6180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f030c076810 tx=0x7f02f800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.187+0000 7f0303400700 1 -- 172.21.15.16:0/3328145667 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02fc09ceb0 con 0x7f030c075510 2024-04-13T03:30:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.420+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f030c02dfa0 con 0x7f030c075510 2024-04-13T03:30:39.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.421+0000 7f0303400700 1 -- 172.21.15.16:0/3328145667 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f02fc09c7b0 con 0x7f030c075510 2024-04-13T03:30:39.423 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:30:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.424+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f02f00a3cc0 msgr2=0x7f02f00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.424+0000 7f0311a00700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f02f00a3cc0 0x7f02f00a6180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f030c076810 tx=0x7f02f800b410 comp rx=0 tx=0).stop 2024-04-13T03:30:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.424+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f030c075510 msgr2=0x7f030c19af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.424+0000 7f0311a00700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f030c075510 0x7f030c19af40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00e9a0 tx=0x7f02fc00ecb0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.425+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 shutdown_connections 2024-04-13T03:30:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.425+0000 7f0311a00700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f02f00a3cc0 0x7f02f00a6180 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.425+0000 7f0311a00700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f030c10a9e0 0x7f030c1a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.425+0000 7f0311a00700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f030c075510 0x7f030c19af40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.425+0000 7f0311a00700 1 --2- 172.21.15.16:0/3328145667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f030c076ae0 0x7f030c19b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.425+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 >> 172.21.15.16:0/3328145667 conn(0x7f030c0fe0b0 msgr2=0x7f030c1068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.425+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 shutdown_connections 2024-04-13T03:30:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:39.425+0000 7f0311a00700 1 -- 172.21.15.16:0/3328145667 wait complete. 2024-04-13T03:30:39.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:39 smithi188 ceph-mon[36398]: purged_snaps scrub starts 2024-04-13T03:30:39.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:39 smithi188 ceph-mon[36398]: purged_snaps scrub ok 2024-04-13T03:30:39.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:39 smithi188 ceph-mon[36398]: osd.11 [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] boot 2024-04-13T03:30:39.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:39 smithi188 ceph-mon[36398]: osdmap e81: 12 total, 12 up, 12 in 2024-04-13T03:30:39.967 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:39 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:39.968 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:39 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3328145667' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-13T03:30:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:39 smithi119 ceph-mon[35835]: purged_snaps scrub starts 2024-04-13T03:30:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:39 smithi119 ceph-mon[35835]: purged_snaps scrub ok 2024-04-13T03:30:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:39 smithi119 ceph-mon[35835]: osd.11 [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] boot 2024-04-13T03:30:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:39 smithi119 ceph-mon[35835]: osdmap e81: 12 total, 12 up, 12 in 2024-04-13T03:30:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:39 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:39 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3328145667' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-13T03:30:40.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:39 smithi016 ceph-mon[31772]: purged_snaps scrub starts 2024-04-13T03:30:40.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:39 smithi016 ceph-mon[31772]: purged_snaps scrub ok 2024-04-13T03:30:40.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:39 smithi016 ceph-mon[31772]: osd.11 [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] boot 2024-04-13T03:30:40.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:39 smithi016 ceph-mon[31772]: osdmap e81: 12 total, 12 up, 12 in 2024-04-13T03:30:40.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:39 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:30:40.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:39 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3328145667' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-13T03:30:40.198 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1712979038,"num_in_osds":12,"osd_in_since":1712979019,"num_remapped_pgs":0} 2024-04-13T03:30:40.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd dump --format=json 2024-04-13T03:30:40.625 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:30:40.970 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: osdmap e82: 12 total, 12 up, 12 in 2024-04-13T03:30:40.970 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:40.970 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: Detected new or changed devices on smithi188 2024-04-13T03:30:40.970 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:40.970 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:40.970 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:40.970 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:40.970 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:40.971 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:40.971 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi188 to 4270M 2024-04-13T03:30:40.971 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:40.971 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:40.971 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:40.971 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:40.971 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: osdmap e82: 12 total, 12 up, 12 in 2024-04-13T03:30:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: Detected new or changed devices on smithi188 2024-04-13T03:30:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi188 to 4270M 2024-04-13T03:30:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: osdmap e82: 12 total, 12 up, 12 in 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: Detected new or changed devices on smithi188 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi188 to 4270M 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:42.021 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:42.021 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:42.021 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:42.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:42.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:42.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:42.933 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:42 smithi016 ceph-mon[31772]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.936+0000 7f06a0400700 1 -- 172.21.15.16:0/3065848064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 msgr2=0x7f069810f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.936+0000 7f06a0400700 1 --2- 172.21.15.16:0/3065848064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 0x7f069810f970 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f069000b0a0 tx=0x7f069000b3b0 comp rx=0 tx=0).stop 2024-04-13T03:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.936+0000 7f06a0400700 1 -- 172.21.15.16:0/3065848064 shutdown_connections 2024-04-13T03:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.936+0000 7f06a0400700 1 --2- 172.21.15.16:0/3065848064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 0x7f069810f970 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.936+0000 7f06a0400700 1 --2- 172.21.15.16:0/3065848064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0698108bc0 0x7f0698108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.936+0000 7f06a0400700 1 --2- 172.21.15.16:0/3065848064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f069806b1e0 0x7f069806b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.936+0000 7f06a0400700 1 -- 172.21.15.16:0/3065848064 >> 172.21.15.16:0/3065848064 conn(0x7f0698077820 msgr2=0x7f0698077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.937+0000 7f06a0400700 1 -- 172.21.15.16:0/3065848064 shutdown_connections 2024-04-13T03:30:42.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.937+0000 7f06a0400700 1 -- 172.21.15.16:0/3065848064 wait complete. 2024-04-13T03:30:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.938+0000 7f06a0400700 1 Processor -- start 2024-04-13T03:30:42.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.938+0000 7f06a0400700 1 -- start start 2024-04-13T03:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f06a0400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f069806b1e0 0x7f0698101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f06a0400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 0x7f0698102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f06a0400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0698108bc0 0x7f0698102850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f06a0400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0698112df0 con 0x7f069806bba0 2024-04-13T03:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f06a0400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0698112c70 con 0x7f069806b1e0 2024-04-13T03:30:42.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f06a0400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0698112f70 con 0x7f0698108bc0 2024-04-13T03:30:42.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f069d000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 0x7f0698102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f069d000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 0x7f0698102310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38522/0 (socket says 172.21.15.16:38522) 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f069d000700 1 -- 172.21.15.16:0/1907137945 learned_addr learned my addr 172.21.15.16:0/1907137945 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f069e400700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0698108bc0 0x7f0698102850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f069da00700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f069806b1e0 0x7f0698101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.939+0000 7f069d000700 1 -- 172.21.15.16:0/1907137945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0698108bc0 msgr2=0x7f0698102850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f069d000700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0698108bc0 0x7f0698102850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f069d000700 1 -- 172.21.15.16:0/1907137945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f069806b1e0 msgr2=0x7f0698101dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f069d000700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f069806b1e0 0x7f0698101dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f069d000700 1 -- 172.21.15.16:0/1907137945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06981a7710 con 0x7f069806bba0 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f069e400700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0698108bc0 0x7f0698102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:30:42.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f069d000700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 0x7f0698102310 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f068400eb30 tx=0x7f068400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f0696000700 1 -- 172.21.15.16:0/1907137945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068400cbd0 con 0x7f069806bba0 2024-04-13T03:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f0696000700 1 -- 172.21.15.16:0/1907137945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0684004500 con 0x7f069806bba0 2024-04-13T03:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06981a7a00 con 0x7f069806bba0 2024-04-13T03:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f0696000700 1 -- 172.21.15.16:0/1907137945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684010430 con 0x7f069806bba0 2024-04-13T03:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.940+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06981a7f90 con 0x7f069806bba0 2024-04-13T03:30:42.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.942+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0698104210 con 0x7f069806bba0 2024-04-13T03:30:42.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.944+0000 7f0696000700 1 -- 172.21.15.16:0/1907137945 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0684003f00 con 0x7f069806bba0 2024-04-13T03:30:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.945+0000 7f0696000700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f06800a3cc0 0x7f06800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.945+0000 7f0696000700 1 -- 172.21.15.16:0/1907137945 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f06840d02b0 con 0x7f069806bba0 2024-04-13T03:30:42.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.945+0000 7f069da00700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f06800a3cc0 0x7f06800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:42.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.946+0000 7f069da00700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f06800a3cc0 0x7f06800a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f068c009be0 tx=0x7f068c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:42.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:42.950+0000 7f0696000700 1 -- 172.21.15.16:0/1907137945 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f068409d090 con 0x7f069806bba0 2024-04-13T03:30:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:42 smithi119 ceph-mon[35835]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:42 smithi188 ceph-mon[36398]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:43.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.185+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f069802dfa0 con 0x7f069806bba0 2024-04-13T03:30:43.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.190+0000 7f0696000700 1 -- 172.21.15.16:0/1907137945 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18488 (secure 0 0 0) 0x7f068409c990 con 0x7f069806bba0 2024-04-13T03:30:43.191 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:30:43.192 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:30:39.665995+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:30:43.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f06800a3cc0 msgr2=0x7f06800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:43.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f06800a3cc0 0x7f06800a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f068c009be0 tx=0x7f068c009260 comp rx=0 tx=0).stop 2024-04-13T03:30:43.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 msgr2=0x7f0698102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:43.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 0x7f0698102310 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f068400eb30 tx=0x7f068400ee40 comp rx=0 tx=0).stop 2024-04-13T03:30:43.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 shutdown_connections 2024-04-13T03:30:43.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f06800a3cc0 0x7f06800a6180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:43.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0698108bc0 0x7f0698102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:43.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f069806b1e0 0x7f0698101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:43.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 --2- 172.21.15.16:0/1907137945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f069806bba0 0x7f0698102310 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:43.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 >> 172.21.15.16:0/1907137945 conn(0x7f0698077820 msgr2=0x7f0698100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:43.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.193+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 shutdown_connections 2024-04-13T03:30:43.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:43.194+0000 7f06a0400700 1 -- 172.21.15.16:0/1907137945 wait complete. 2024-04-13T03:30:43.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:43 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1907137945' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:30:44.030 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-13T03:30:44.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-13T03:30:44.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:43 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1907137945' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:30:44.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:43 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1907137945' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:30:44.512 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:30:44.984 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:44 smithi016 ceph-mon[31772]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:44 smithi119 ceph-mon[35835]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:45.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:44 smithi188 ceph-mon[36398]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:46.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.752+0000 7f5009400700 1 -- 172.21.15.16:0/3773020012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004103530 msgr2=0x7f500410f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.752+0000 7f5009400700 1 --2- 172.21.15.16:0/3773020012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004103530 0x7f500410f950 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00b0a0 tx=0x7f4ffc00b3b0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.752+0000 7f5009400700 1 -- 172.21.15.16:0/3773020012 shutdown_connections 2024-04-13T03:30:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.752+0000 7f5009400700 1 --2- 172.21.15.16:0/3773020012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004103530 0x7f500410f950 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.752+0000 7f5009400700 1 --2- 172.21.15.16:0/3773020012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5004108a80 0x7f5004108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.752+0000 7f5009400700 1 --2- 172.21.15.16:0/3773020012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5004102b70 0x7f5004102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.752+0000 7f5009400700 1 -- 172.21.15.16:0/3773020012 >> 172.21.15.16:0/3773020012 conn(0x7f5004079c70 msgr2=0x7f500407a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.752+0000 7f5009400700 1 -- 172.21.15.16:0/3773020012 shutdown_connections 2024-04-13T03:30:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.752+0000 7f5009400700 1 -- 172.21.15.16:0/3773020012 wait complete. 2024-04-13T03:30:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.753+0000 7f5009400700 1 Processor -- start 2024-04-13T03:30:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.753+0000 7f5009400700 1 -- start start 2024-04-13T03:30:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5009400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004102b70 0x7f5004199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:46.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5009400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5004103530 0x7f5004199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5009400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5004108a80 0x7f500419dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5009400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5004112e10 con 0x7f5004102b70 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5009400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5004112c90 con 0x7f5004103530 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5009400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5004112f90 con 0x7f5004108a80 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5002800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004102b70 0x7f5004199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5002800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004102b70 0x7f5004199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38534/0 (socket says 172.21.15.16:38534) 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5002800700 1 -- 172.21.15.16:0/3508193613 learned_addr learned my addr 172.21.15.16:0/3508193613 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5001e00700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5004103530 0x7f5004199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.754+0000 7f5003200700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5004108a80 0x7f500419dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f5002800700 1 -- 172.21.15.16:0/3508193613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5004108a80 msgr2=0x7f500419dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f5002800700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5004108a80 0x7f500419dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f5002800700 1 -- 172.21.15.16:0/3508193613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5004103530 msgr2=0x7f5004199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f5002800700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5004103530 0x7f5004199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f5002800700 1 -- 172.21.15.16:0/3508193613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f500419e210 con 0x7f5004102b70 2024-04-13T03:30:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f5002800700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004102b70 0x7f5004199200 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400e9c0 tx=0x7f4ff400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f4ffb400700 1 -- 172.21.15.16:0/3508193613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400cbd0 con 0x7f5004102b70 2024-04-13T03:30:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f4ffb400700 1 -- 172.21.15.16:0/3508193613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ff400cd70 con 0x7f5004102b70 2024-04-13T03:30:46.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f500419e500 con 0x7f5004102b70 2024-04-13T03:30:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.755+0000 7f4ffb400700 1 -- 172.21.15.16:0/3508193613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4010430 con 0x7f5004102b70 2024-04-13T03:30:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.756+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50041a5d80 con 0x7f5004102b70 2024-04-13T03:30:46.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.757+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50040697a0 con 0x7f5004102b70 2024-04-13T03:30:46.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.759+0000 7f4ffb400700 1 -- 172.21.15.16:0/3508193613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4ff4004750 con 0x7f5004102b70 2024-04-13T03:30:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.760+0000 7f4ffb400700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fe80a3cc0 0x7f4fe80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.761+0000 7f4ffb400700 1 -- 172.21.15.16:0/3508193613 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4ff40d01b0 con 0x7f5004102b70 2024-04-13T03:30:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.761+0000 7f5001e00700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fe80a3cc0 0x7f4fe80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:46.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.761+0000 7f5001e00700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fe80a3cc0 0x7f4fe80a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009680 tx=0x7f4ff0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:46.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.765+0000 7f4ffb400700 1 -- 172.21.15.16:0/3508193613 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ff409cf90 con 0x7f5004102b70 2024-04-13T03:30:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.994+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f500402dfa0 con 0x7f5004102b70 2024-04-13T03:30:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.994+0000 7f4ffb400700 1 -- 172.21.15.16:0/3508193613 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f4ff4085e20 con 0x7f5004102b70 2024-04-13T03:30:46.996 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:30:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.997+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fe80a3cc0 msgr2=0x7f4fe80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.997+0000 7f5009400700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fe80a3cc0 0x7f4fe80a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009680 tx=0x7f4ff0009260 comp rx=0 tx=0).stop 2024-04-13T03:30:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.997+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004102b70 msgr2=0x7f5004199200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.997+0000 7f5009400700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004102b70 0x7f5004199200 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400e9c0 tx=0x7f4ff400ecd0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.998+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 shutdown_connections 2024-04-13T03:30:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.998+0000 7f5009400700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fe80a3cc0 0x7f4fe80a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.998+0000 7f5009400700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5004108a80 0x7f500419dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.998+0000 7f5009400700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5004102b70 0x7f5004199200 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.998+0000 7f5009400700 1 --2- 172.21.15.16:0/3508193613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5004103530 0x7f5004199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.998+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 >> 172.21.15.16:0/3508193613 conn(0x7f5004079c70 msgr2=0x7f5004100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.998+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 shutdown_connections 2024-04-13T03:30:47.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:46.998+0000 7f5009400700 1 -- 172.21.15.16:0/3508193613 wait complete. 2024-04-13T03:30:47.018 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:46 smithi016 ceph-mon[31772]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:47.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:46 smithi119 ceph-mon[35835]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:46 smithi188 ceph-mon[36398]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:47.861 INFO:tasks.cephadm:Adding mds.a on smithi016 2024-04-13T03:30:47.861 INFO:tasks.cephadm:Adding mds.d on smithi016 2024-04-13T03:30:47.861 INFO:tasks.cephadm:Adding mds.g on smithi016 2024-04-13T03:30:47.861 INFO:tasks.cephadm:Adding mds.j on smithi016 2024-04-13T03:30:47.861 INFO:tasks.cephadm:Adding mds.b on smithi119 2024-04-13T03:30:47.861 INFO:tasks.cephadm:Adding mds.e on smithi119 2024-04-13T03:30:47.861 INFO:tasks.cephadm:Adding mds.h on smithi119 2024-04-13T03:30:47.862 INFO:tasks.cephadm:Adding mds.k on smithi119 2024-04-13T03:30:47.862 INFO:tasks.cephadm:Adding mds.c on smithi188 2024-04-13T03:30:47.862 INFO:tasks.cephadm:Adding mds.f on smithi188 2024-04-13T03:30:47.862 INFO:tasks.cephadm:Adding mds.i on smithi188 2024-04-13T03:30:47.862 INFO:tasks.cephadm:Adding mds.l on smithi188 2024-04-13T03:30:47.862 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph orch apply mds all '12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l' 2024-04-13T03:30:48.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:47 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3508193613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:30:48.114 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:30:48.235 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:47 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3508193613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:30:48.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:47 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3508193613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:30:49.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:48 smithi188 ceph-mon[36398]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:48 smithi119 ceph-mon[35835]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:48 smithi016 ceph-mon[31772]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:49.680 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.679+0000 7fa968a15700 1 -- 172.21.15.188:0/1032687462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 msgr2=0x7fa964103080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:49.680 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.679+0000 7fa968a15700 1 --2- 172.21.15.188:0/1032687462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 0x7fa964103080 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa958009990 tx=0x7fa958009ca0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.680 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.679+0000 7fa968a15700 1 -- 172.21.15.188:0/1032687462 shutdown_connections 2024-04-13T03:30:49.680 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.679+0000 7fa968a15700 1 --2- 172.21.15.188:0/1032687462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9641035c0 0x7fa964109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.680 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.679+0000 7fa968a15700 1 --2- 172.21.15.188:0/1032687462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa964101a20 0x7fa964101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.680 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.679+0000 7fa968a15700 1 --2- 172.21.15.188:0/1032687462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 0x7fa964103080 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.680 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.679+0000 7fa968a15700 1 -- 172.21.15.188:0/1032687462 >> 172.21.15.188:0/1032687462 conn(0x7fa9640fcfe0 msgr2=0x7fa9640ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:49.680 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.679+0000 7fa968a15700 1 -- 172.21.15.188:0/1032687462 shutdown_connections 2024-04-13T03:30:49.681 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.679+0000 7fa968a15700 1 -- 172.21.15.188:0/1032687462 wait complete. 2024-04-13T03:30:49.682 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa968a15700 1 Processor -- start 2024-04-13T03:30:49.682 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa968a15700 1 -- start start 2024-04-13T03:30:49.682 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa968a15700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa964101a20 0x7fa964197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa968a15700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 0x7fa964198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa968a15700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9641035c0 0x7fa96419f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa968a15700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa96410d680 con 0x7fa964101a20 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa968a15700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa96410d500 con 0x7fa9641035c0 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa968a15700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa96410d800 con 0x7fa964102c20 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa961d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 0x7fa964198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa961d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 0x7fa964198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:48400/0 (socket says 172.21.15.188:48400) 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa961d9b700 1 -- 172.21.15.188:0/1397653739 learned_addr learned my addr 172.21.15.188:0/1397653739 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa962d9d700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9641035c0 0x7fa96419f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa961d9b700 1 -- 172.21.15.188:0/1397653739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9641035c0 msgr2=0x7fa96419f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa961d9b700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9641035c0 0x7fa96419f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa961d9b700 1 -- 172.21.15.188:0/1397653739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa964101a20 msgr2=0x7fa964197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa961d9b700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa964101a20 0x7fa964197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa961d9b700 1 -- 172.21.15.188:0/1397653739 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96419f930 con 0x7fa964102c20 2024-04-13T03:30:49.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.680+0000 7fa961d9b700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 0x7fa964198220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa9580098e0 tx=0x7fa958009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:49.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.681+0000 7fa9537fe700 1 -- 172.21.15.188:0/1397653739 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa958027070 con 0x7fa964102c20 2024-04-13T03:30:49.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.681+0000 7fa9537fe700 1 -- 172.21.15.188:0/1397653739 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa958004150 con 0x7fa964102c20 2024-04-13T03:30:49.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.681+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96419fbc0 con 0x7fa964102c20 2024-04-13T03:30:49.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.681+0000 7fa9537fe700 1 -- 172.21.15.188:0/1397653739 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95800f460 con 0x7fa964102c20 2024-04-13T03:30:49.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.681+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9641a0010 con 0x7fa964102c20 2024-04-13T03:30:49.684 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.682+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa96402dfa0 con 0x7fa964102c20 2024-04-13T03:30:49.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.683+0000 7fa9537fe700 1 -- 172.21.15.188:0/1397653739 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa95802a070 con 0x7fa964102c20 2024-04-13T03:30:49.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.683+0000 7fa9537fe700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa94c0a3c40 0x7fa94c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:49.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.683+0000 7fa9537fe700 1 -- 172.21.15.188:0/1397653739 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa9580d0ce0 con 0x7fa964102c20 2024-04-13T03:30:49.685 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.684+0000 7fa96259c700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa94c0a3c40 0x7fa94c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:49.686 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.686+0000 7fa96259c700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa94c0a3c40 0x7fa94c0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa9641991c0 tx=0x7fa954009470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:49.688 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.688+0000 7fa9537fe700 1 -- 172.21.15.188:0/1397653739 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa958013070 con 0x7fa964102c20 2024-04-13T03:30:49.896 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.895+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}) -- 0x7fa964062ff0 con 0x7fa94c0a3c40 2024-04-13T03:30:49.913 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.912+0000 7fa9537fe700 1 -- 172.21.15.188:0/1397653739 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fa964062ff0 con 0x7fa94c0a3c40 2024-04-13T03:30:49.914 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mds.all update... 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa94c0a3c40 msgr2=0x7fa94c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa94c0a3c40 0x7fa94c0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa9641991c0 tx=0x7fa954009470 comp rx=0 tx=0).stop 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 msgr2=0x7fa964198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 0x7fa964198220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa9580098e0 tx=0x7fa958009ee0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 shutdown_connections 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa94c0a3c40 0x7fa94c0a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9641035c0 0x7fa96419f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa964101a20 0x7fa964197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 --2- 172.21.15.188:0/1397653739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa964102c20 0x7fa964198220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 >> 172.21.15.188:0/1397653739 conn(0x7fa9640fcfe0 msgr2=0x7fa964105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 shutdown_connections 2024-04-13T03:30:49.916 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-13T03:30:49.915+0000 7fa968a15700 1 -- 172.21.15.188:0/1397653739 wait complete. 2024-04-13T03:30:50.628 DEBUG:teuthology.orchestra.run.smithi016:mds.a> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.a.service 2024-04-13T03:30:50.633 DEBUG:teuthology.orchestra.run.smithi016:mds.d> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.d.service 2024-04-13T03:30:50.678 DEBUG:teuthology.orchestra.run.smithi016:mds.g> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.g.service 2024-04-13T03:30:50.722 DEBUG:teuthology.orchestra.run.smithi016:mds.j> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.j.service 2024-04-13T03:30:50.771 DEBUG:teuthology.orchestra.run.smithi119:mds.b> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.b.service 2024-04-13T03:30:50.774 DEBUG:teuthology.orchestra.run.smithi119:mds.e> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.e.service 2024-04-13T03:30:50.817 DEBUG:teuthology.orchestra.run.smithi119:mds.h> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.h.service 2024-04-13T03:30:50.860 DEBUG:teuthology.orchestra.run.smithi119:mds.k> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.k.service 2024-04-13T03:30:50.903 DEBUG:teuthology.orchestra.run.smithi188:mds.c> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.c.service 2024-04-13T03:30:50.905 DEBUG:teuthology.orchestra.run.smithi188:mds.f> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.f.service 2024-04-13T03:30:50.948 DEBUG:teuthology.orchestra.run.smithi188:mds.i> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.i.service 2024-04-13T03:30:50.991 DEBUG:teuthology.orchestra.run.smithi188:mds.l> sudo journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.l.service 2024-04-13T03:30:51.034 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-13T03:30:51.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:30:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:50 smithi119 ceph-mon[35835]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:50 smithi119 ceph-mon[35835]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:30:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:50 smithi119 ceph-mon[35835]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-13T03:30:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:50 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:50 smithi016 ceph-mon[31772]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:50 smithi016 ceph-mon[31772]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:30:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:50 smithi016 ceph-mon[31772]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-13T03:30:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:50 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:51.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.291+0000 7ff8dce00640 1 -- 172.21.15.16:0/3401357288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81028c0 msgr2=0x7ff8d8104cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.291+0000 7ff8dce00640 1 --2- 172.21.15.16:0/3401357288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81028c0 0x7ff8d8104cb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009860 tx=0x7ff8c002bb70 comp rx=0 tx=0).stop 2024-04-13T03:30:51.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.291+0000 7ff8dce00640 1 -- 172.21.15.16:0/3401357288 shutdown_connections 2024-04-13T03:30:51.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.291+0000 7ff8dce00640 1 --2- 172.21.15.16:0/3401357288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d8107b20 0x7ff8d8109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.291+0000 7ff8dce00640 1 --2- 172.21.15.16:0/3401357288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8d81051f0 0x7ff8d81075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.291+0000 7ff8dce00640 1 --2- 172.21.15.16:0/3401357288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81028c0 0x7ff8d8104cb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.291+0000 7ff8dce00640 1 -- 172.21.15.16:0/3401357288 >> 172.21.15.16:0/3401357288 conn(0x7ff8d80fc290 msgr2=0x7ff8d80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.291+0000 7ff8dce00640 1 -- 172.21.15.16:0/3401357288 shutdown_connections 2024-04-13T03:30:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.291+0000 7ff8dce00640 1 -- 172.21.15.16:0/3401357288 wait complete. 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.292+0000 7ff8dce00640 1 Processor -- start 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.292+0000 7ff8dce00640 1 -- start start 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.292+0000 7ff8dce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d81028c0 0x7ff8d819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.292+0000 7ff8dce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81051f0 0x7ff8d819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.292+0000 7ff8dce00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8d8107b20 0x7ff8d81a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.292+0000 7ff8dce00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d810db10 con 0x7ff8d81051f0 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.292+0000 7ff8dce00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d810d990 con 0x7ff8d81028c0 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.292+0000 7ff8dce00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d810dc90 con 0x7ff8d8107b20 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81051f0 0x7ff8d819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:51.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8d8107b20 0x7ff8d81a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d7400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d81028c0 0x7ff8d819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81051f0 0x7ff8d819a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36142/0 (socket says 172.21.15.16:36142) 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d6a00640 1 -- 172.21.15.16:0/2503073860 learned_addr learned my addr 172.21.15.16:0/2503073860 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8d8107b20 0x7ff8d81a19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:46060/0 (socket says 172.21.15.16:46060) 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d6a00640 1 -- 172.21.15.16:0/2503073860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8d8107b20 msgr2=0x7ff8d81a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d6a00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8d8107b20 0x7ff8d81a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d6a00640 1 -- 172.21.15.16:0/2503073860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d81028c0 msgr2=0x7ff8d819a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d6a00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d81028c0 0x7ff8d819a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d6a00640 1 -- 172.21.15.16:0/2503073860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d81a2040 con 0x7ff8d81051f0 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d7400640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d81028c0 0x7ff8d819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.293+0000 7ff8d6a00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81051f0 0x7ff8d819a960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800eb90 tx=0x7ff8c800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.294+0000 7ff8cfe00640 1 -- 172.21.15.16:0/2503073860 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8010630 con 0x7ff8d81051f0 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.294+0000 7ff8cfe00640 1 -- 172.21.15.16:0/2503073860 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8c800ee90 con 0x7ff8d81051f0 2024-04-13T03:30:51.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.294+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d81a2330 con 0x7ff8d81051f0 2024-04-13T03:30:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.294+0000 7ff8cfe00640 1 -- 172.21.15.16:0/2503073860 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8018b90 con 0x7ff8d81051f0 2024-04-13T03:30:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.294+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d81a2880 con 0x7ff8d81051f0 2024-04-13T03:30:51.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.295+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d8109670 con 0x7ff8d81051f0 2024-04-13T03:30:51.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.297+0000 7ff8cfe00640 1 -- 172.21.15.16:0/2503073860 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff8c80107d0 con 0x7ff8d81051f0 2024-04-13T03:30:51.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.298+0000 7ff8cfe00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff8bc0a3c00 0x7ff8bc0a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:51.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.299+0000 7ff8cfe00640 1 -- 172.21.15.16:0/2503073860 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff8c80d0970 con 0x7ff8d81051f0 2024-04-13T03:30:51.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.299+0000 7ff8d7400640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff8bc0a3c00 0x7ff8bc0a60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:51.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.300+0000 7ff8d7400640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff8bc0a3c00 0x7ff8bc0a60c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff8c002c180 tx=0x7ff8c002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:51.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.302+0000 7ff8cfe00640 1 -- 172.21.15.16:0/2503073860 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8c809d100 con 0x7ff8d81051f0 2024-04-13T03:30:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:50 smithi188 ceph-mon[36398]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:50 smithi188 ceph-mon[36398]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:30:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:50 smithi188 ceph-mon[36398]: Saving service mds.all spec with placement smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-13T03:30:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:50 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.511+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8d802d830 con 0x7ff8d81051f0 2024-04-13T03:30:51.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.514+0000 7ff8cfe00640 1 -- 172.21.15.16:0/2503073860 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18488 (secure 0 0 0) 0x7ff8c80a1400 con 0x7ff8d81051f0 2024-04-13T03:30:51.516 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:30:51.516 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:30:39.665995+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:30:51.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff8bc0a3c00 msgr2=0x7ff8bc0a60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff8bc0a3c00 0x7ff8bc0a60c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff8c002c180 tx=0x7ff8c002e040 comp rx=0 tx=0).stop 2024-04-13T03:30:51.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81051f0 msgr2=0x7ff8d819a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81051f0 0x7ff8d819a960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800eb90 tx=0x7ff8c800c6a0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 shutdown_connections 2024-04-13T03:30:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff8bc0a3c00 0x7ff8bc0a60c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8d8107b20 0x7ff8d81a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8d81051f0 0x7ff8d819a960 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 --2- 172.21.15.16:0/2503073860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8d81028c0 0x7ff8d819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 >> 172.21.15.16:0/2503073860 conn(0x7ff8d80fc290 msgr2=0x7ff8d8102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 shutdown_connections 2024-04-13T03:30:51.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.517+0000 7ff8dce00640 1 -- 172.21.15.16:0/2503073860 wait complete. 2024-04-13T03:30:51.539 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-13T03:30:51.539 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:30:51.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.769+0000 7fd37a400640 1 -- 172.21.15.16:0/2050318195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd374109730 msgr2=0x7fd374109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.769+0000 7fd37a400640 1 --2- 172.21.15.16:0/2050318195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd374109730 0x7fd374109bb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd35c009ca0 tx=0x7fd35c02c0f0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.769+0000 7fd37a400640 1 -- 172.21.15.16:0/2050318195 shutdown_connections 2024-04-13T03:30:51.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.769+0000 7fd37a400640 1 --2- 172.21.15.16:0/2050318195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd37410a0f0 0x7fd374110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.769+0000 7fd37a400640 1 --2- 172.21.15.16:0/2050318195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd374109730 0x7fd374109bb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.769+0000 7fd37a400640 1 --2- 172.21.15.16:0/2050318195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd374108540 0x7fd374108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.769+0000 7fd37a400640 1 -- 172.21.15.16:0/2050318195 >> 172.21.15.16:0/2050318195 conn(0x7fd3740774f0 msgr2=0x7fd374079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:51.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.769+0000 7fd37a400640 1 -- 172.21.15.16:0/2050318195 shutdown_connections 2024-04-13T03:30:51.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.769+0000 7fd37a400640 1 -- 172.21.15.16:0/2050318195 wait complete. 2024-04-13T03:30:51.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.770+0000 7fd37a400640 1 Processor -- start 2024-04-13T03:30:51.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.770+0000 7fd37a400640 1 -- start start 2024-04-13T03:30:51.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.770+0000 7fd37a400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd374108540 0x7fd37419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:51.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.770+0000 7fd37a400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd374109730 0x7fd37419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:51.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd37a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd37410a0f0 0x7fd3741a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:51.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd37a400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd3741141d0 con 0x7fd37410a0f0 2024-04-13T03:30:51.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd37a400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd374114050 con 0x7fd374109730 2024-04-13T03:30:51.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd37a400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd374114350 con 0x7fd374108540 2024-04-13T03:30:51.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd379000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd374108540 0x7fd37419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd379a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd37410a0f0 0x7fd3741a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd379000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd374108540 0x7fd37419e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:46076/0 (socket says 172.21.15.16:46076) 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd379000640 1 -- 172.21.15.16:0/147098343 learned_addr learned my addr 172.21.15.16:0/147098343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd36be00640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd374109730 0x7fd37419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd379000640 1 -- 172.21.15.16:0/147098343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd374109730 msgr2=0x7fd37419ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd379000640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd374109730 0x7fd37419ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd379000640 1 -- 172.21.15.16:0/147098343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd37410a0f0 msgr2=0x7fd3741a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd379000640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd37410a0f0 0x7fd3741a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.771+0000 7fd379000640 1 -- 172.21.15.16:0/147098343 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3741a63f0 con 0x7fd374108540 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.772+0000 7fd379000640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd374108540 0x7fd37419e7d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd36400e9c0 tx=0x7fd36400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.772+0000 7fd372000640 1 -- 172.21.15.16:0/147098343 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36400cd80 con 0x7fd374108540 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.772+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3741a66e0 con 0x7fd374108540 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.772+0000 7fd372000640 1 -- 172.21.15.16:0/147098343 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3640035f0 con 0x7fd374108540 2024-04-13T03:30:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.772+0000 7fd372000640 1 -- 172.21.15.16:0/147098343 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364010690 con 0x7fd374108540 2024-04-13T03:30:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.772+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3741a6c50 con 0x7fd374108540 2024-04-13T03:30:51.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.773+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd37410f860 con 0x7fd374108540 2024-04-13T03:30:51.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.775+0000 7fd372000640 1 -- 172.21.15.16:0/147098343 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd3640037a0 con 0x7fd374108540 2024-04-13T03:30:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.776+0000 7fd372000640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3580a3bb0 0x7fd3580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:51.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.777+0000 7fd372000640 1 -- 172.21.15.16:0/147098343 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd3640d0360 con 0x7fd374108540 2024-04-13T03:30:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.777+0000 7fd36be00640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3580a3bb0 0x7fd3580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:51.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.777+0000 7fd36be00640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3580a3bb0 0x7fd3580a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd35c02c700 tx=0x7fd35c0323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:51.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.785+0000 7fd372000640 1 -- 172.21.15.16:0/147098343 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd36409d1c0 con 0x7fd374108540 2024-04-13T03:30:51.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.993+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd37402d890 con 0x7fd374108540 2024-04-13T03:30:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.994+0000 7fd372000640 1 -- 172.21.15.16:0/147098343 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fd36409cac0 con 0x7fd374108540 2024-04-13T03:30:51.995 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.996+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3580a3bb0 msgr2=0x7fd3580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.996+0000 7fd37a400640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3580a3bb0 0x7fd3580a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd35c02c700 tx=0x7fd35c0323c0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.996+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd374108540 msgr2=0x7fd37419e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.996+0000 7fd37a400640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd374108540 0x7fd37419e7d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd36400e9c0 tx=0x7fd36400ee80 comp rx=0 tx=0).stop 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.996+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 shutdown_connections 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.997+0000 7fd37a400640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3580a3bb0 0x7fd3580a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.997+0000 7fd37a400640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd37410a0f0 0x7fd3741a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.997+0000 7fd37a400640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd374109730 0x7fd37419ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.997+0000 7fd37a400640 1 --2- 172.21.15.16:0/147098343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd374108540 0x7fd37419e7d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.997+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 >> 172.21.15.16:0/147098343 conn(0x7fd3740774f0 msgr2=0x7fd37410ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.997+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 shutdown_connections 2024-04-13T03:30:51.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:51.997+0000 7fd37a400640 1 -- 172.21.15.16:0/147098343 wait complete. 2024-04-13T03:30:52.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:30:52.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:52.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:52.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:52.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:30:52.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:30:52.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:51 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:52.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:51 smithi188 ceph-mon[36398]: Deploying daemon mds.l on smithi188 2024-04-13T03:30:52.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:51 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2503073860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:30:52.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.244+0000 7f828f200640 1 -- 172.21.15.16:0/1447402976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8288101bf0 msgr2=0x7f8288102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:52.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.244+0000 7f828f200640 1 --2- 172.21.15.16:0/1447402976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8288101bf0 0x7f8288102070 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8280009ca0 tx=0x7f828002c190 comp rx=0 tx=0).stop 2024-04-13T03:30:52.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.245+0000 7f828f200640 1 -- 172.21.15.16:0/1447402976 shutdown_connections 2024-04-13T03:30:52.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.245+0000 7f828f200640 1 --2- 172.21.15.16:0/1447402976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8288101bf0 0x7f8288102070 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:52.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.245+0000 7f828f200640 1 --2- 172.21.15.16:0/1447402976 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101250 0x7f82881016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:52.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.245+0000 7f828f200640 1 --2- 172.21.15.16:0/1447402976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82881008f0 0x7f8288100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:52.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.245+0000 7f828f200640 1 -- 172.21.15.16:0/1447402976 >> 172.21.15.16:0/1447402976 conn(0x7f82880fc4b0 msgr2=0x7f82880fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:52.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.245+0000 7f828f200640 1 -- 172.21.15.16:0/1447402976 shutdown_connections 2024-04-13T03:30:52.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.245+0000 7f828f200640 1 -- 172.21.15.16:0/1447402976 wait complete. 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f828f200640 1 Processor -- start 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f828f200640 1 -- start start 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f828f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82881008f0 0x7f828819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f828f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101250 0x7f828819a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f828f200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8288101bf0 0x7f82881a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f828f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f828810fe90 con 0x7f82881008f0 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f828f200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f828810fd10 con 0x7f8288101bf0 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f828f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8288110010 con 0x7f8288101250 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f8287e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82881008f0 0x7f828819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:52.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f8287e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82881008f0 0x7f828819a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36186/0 (socket says 172.21.15.16:36186) 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.246+0000 7f8287e00640 1 -- 172.21.15.16:0/4037565170 learned_addr learned my addr 172.21.15.16:0/4037565170 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f828d200640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8288101bf0 0x7f82881a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f827fe00640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101250 0x7f828819a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f8287e00640 1 -- 172.21.15.16:0/4037565170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101250 msgr2=0x7f828819a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f8287e00640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101250 0x7f828819a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f8287e00640 1 -- 172.21.15.16:0/4037565170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8288101bf0 msgr2=0x7f82881a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f8287e00640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8288101bf0 0x7f82881a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f8287e00640 1 -- 172.21.15.16:0/4037565170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82881a20b0 con 0x7f82881008f0 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f828d200640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8288101bf0 0x7f82881a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f8287e00640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82881008f0 0x7f828819a490 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f827800ea60 tx=0x7f827800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.247+0000 7f8285600640 1 -- 172.21.15.16:0/4037565170 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827800cde0 con 0x7f82881008f0 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.248+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82881a23a0 con 0x7f82881008f0 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.248+0000 7f8285600640 1 -- 172.21.15.16:0/4037565170 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82780035f0 con 0x7f82881008f0 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.248+0000 7f8285600640 1 -- 172.21.15.16:0/4037565170 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8278010670 con 0x7f82881008f0 2024-04-13T03:30:52.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.248+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82881a2940 con 0x7f82881008f0 2024-04-13T03:30:52.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.249+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f828810b7a0 con 0x7f82881008f0 2024-04-13T03:30:52.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.251+0000 7f8285600640 1 -- 172.21.15.16:0/4037565170 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f82780037a0 con 0x7f82881008f0 2024-04-13T03:30:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.253+0000 7f8285600640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f826c0a3bb0 0x7f826c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.253+0000 7f8285600640 1 -- 172.21.15.16:0/4037565170 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f82780d02b0 con 0x7f82881008f0 2024-04-13T03:30:52.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.254+0000 7f827fe00640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f826c0a3bb0 0x7f826c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:52.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.254+0000 7f827fe00640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f826c0a3bb0 0x7f826c0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f828819b9a0 tx=0x7f8270005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:52.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.258+0000 7f8285600640 1 -- 172.21.15.16:0/4037565170 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f827809d090 con 0x7f82881008f0 2024-04-13T03:30:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:30:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:30:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:51 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:51 smithi119 ceph-mon[35835]: Deploying daemon mds.l on smithi188 2024-04-13T03:30:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:51 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2503073860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:30:52.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:30:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:30:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:51 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:51 smithi016 ceph-mon[31772]: Deploying daemon mds.l on smithi188 2024-04-13T03:30:52.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:51 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2503073860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:30:52.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.508+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f828802d5e0 con 0x7f82881008f0 2024-04-13T03:30:52.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.509+0000 7f8285600640 1 -- 172.21.15.16:0/4037565170 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f827809c990 con 0x7f82881008f0 2024-04-13T03:30:52.511 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:30:52.511 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-13T03:30:52.511 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 1 2024-04-13T03:30:52.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.511+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f826c0a3bb0 msgr2=0x7f826c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:52.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f826c0a3bb0 0x7f826c0a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f828819b9a0 tx=0x7f8270005f90 comp rx=0 tx=0).stop 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82881008f0 msgr2=0x7f828819a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82881008f0 0x7f828819a490 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f827800ea60 tx=0x7f827800ef20 comp rx=0 tx=0).stop 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 shutdown_connections 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f826c0a3bb0 0x7f826c0a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8288101bf0 0x7f82881a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8288101250 0x7f828819a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 --2- 172.21.15.16:0/4037565170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82881008f0 0x7f828819a490 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 >> 172.21.15.16:0/4037565170 conn(0x7f82880fc4b0 msgr2=0x7f82880fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 shutdown_connections 2024-04-13T03:30:52.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:52.512+0000 7f828f200640 1 -- 172.21.15.16:0/4037565170 wait complete. 2024-04-13T03:30:53.118 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:52 smithi188 ceph-mon[36398]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:53.118 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:52 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/147098343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:30:53.118 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:52 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4037565170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:30:53.312 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:52 smithi016 ceph-mon[31772]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:53.312 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:52 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/147098343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:30:53.312 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:52 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4037565170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:30:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:52 smithi119 ceph-mon[35835]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:52 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/147098343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:30:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:52 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4037565170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:30:53.372 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:30:53 smithi188 systemd[1]: Starting Ceph mds.l for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:30:53.781 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:30:53 smithi188 podman[66790]: 2024-04-13 03:30:53.414385221 +0000 UTC m=+0.017235441 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:30:53.781 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:30:53 smithi188 podman[66790]: 2024-04-13 03:30:53.536577641 +0000 UTC m=+0.139427852 container create ad4dc24fcb9779a194092135dec14747953610fba512d528aa75524c00a9590b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-13T03:30:53.781 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:30:53 smithi188 podman[66790]: 2024-04-13 03:30:53.781317948 +0000 UTC m=+0.384168175 container init ad4dc24fcb9779a194092135dec14747953610fba512d528aa75524c00a9590b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-13T03:30:54.122 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:30:53 smithi188 podman[66790]: 2024-04-13 03:30:53.784043098 +0000 UTC m=+0.386893311 container start ad4dc24fcb9779a194092135dec14747953610fba512d528aa75524c00a9590b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409) 2024-04-13T03:30:54.122 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:30:53 smithi188 ceph-mds[66809]: -- 172.21.15.188:0/1262336808 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5604715cba40 con 0x56047159f000 2024-04-13T03:30:54.123 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:30:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l[66805]: starting mds.l at 2024-04-13T03:30:54.123 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:30:53 smithi188 bash[66790]: ad4dc24fcb9779a194092135dec14747953610fba512d528aa75524c00a9590b 2024-04-13T03:30:54.123 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:30:53 smithi188 systemd[1]: Started Ceph mds.l for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:30:54.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:30:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.770+0000 7f1ee7200640 1 -- 172.21.15.16:0/3619844026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee0076980 msgr2=0x7f1ee0076da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.770+0000 7f1ee7200640 1 --2- 172.21.15.16:0/3619844026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee0076980 0x7f1ee0076da0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4009900 tx=0x7f1ed402bc30 comp rx=0 tx=0).stop 2024-04-13T03:30:54.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.770+0000 7f1ee7200640 1 -- 172.21.15.16:0/3619844026 shutdown_connections 2024-04-13T03:30:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.770+0000 7f1ee7200640 1 --2- 172.21.15.16:0/3619844026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ee0078560 0x7f1ee0078e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.770+0000 7f1ee7200640 1 --2- 172.21.15.16:0/3619844026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ee0077bc0 0x7f1ee0078020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.770+0000 7f1ee7200640 1 --2- 172.21.15.16:0/3619844026 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee0076980 0x7f1ee0076da0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.770+0000 7f1ee7200640 1 -- 172.21.15.16:0/3619844026 >> 172.21.15.16:0/3619844026 conn(0x7f1ee00ff5d0 msgr2=0x7f1ee01019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.770+0000 7f1ee7200640 1 -- 172.21.15.16:0/3619844026 shutdown_connections 2024-04-13T03:30:54.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.770+0000 7f1ee7200640 1 -- 172.21.15.16:0/3619844026 wait complete. 2024-04-13T03:30:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.771+0000 7f1ee7200640 1 Processor -- start 2024-04-13T03:30:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.771+0000 7f1ee7200640 1 -- start start 2024-04-13T03:30:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee7200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ee0076980 0x7f1ee019e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee0077bc0 0x7f1ee019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:54.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ee0078560 0x7f1ee01a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee7200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee0114110 con 0x7f1ee0077bc0 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee7200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee0113f90 con 0x7f1ee0078560 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee7200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee0114290 con 0x7f1ee0076980 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee5e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ee0076980 0x7f1ee019e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee0077bc0 0x7f1ee019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee5e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ee0076980 0x7f1ee019e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:46116/0 (socket says 172.21.15.16:46116) 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee5e00640 1 -- 172.21.15.16:0/488373013 learned_addr learned my addr 172.21.15.16:0/488373013 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee6800640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ee0078560 0x7f1ee01a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee6800640 1 -- 172.21.15.16:0/488373013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ee0076980 msgr2=0x7f1ee019e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee6800640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ee0076980 0x7f1ee019e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee6800640 1 -- 172.21.15.16:0/488373013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee0077bc0 msgr2=0x7f1ee019ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee6800640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee0077bc0 0x7f1ee019ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee6800640 1 -- 172.21.15.16:0/488373013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee01a6330 con 0x7f1ee0078560 2024-04-13T03:30:54.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.772+0000 7f1ee5e00640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ee0076980 0x7f1ee019e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:30:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.773+0000 7f1ee6800640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ee0078560 0x7f1ee01a5cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009fd0 tx=0x7f1edc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.773+0000 7f1edaa00640 1 -- 172.21.15.16:0/488373013 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc00ce20 con 0x7f1ee0078560 2024-04-13T03:30:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.773+0000 7f1edaa00640 1 -- 172.21.15.16:0/488373013 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1edc005020 con 0x7f1ee0078560 2024-04-13T03:30:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.773+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee01a6620 con 0x7f1ee0078560 2024-04-13T03:30:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.773+0000 7f1edaa00640 1 -- 172.21.15.16:0/488373013 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc010720 con 0x7f1ee0078560 2024-04-13T03:30:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.773+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee01a6b70 con 0x7f1ee0078560 2024-04-13T03:30:54.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.774+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ee0110b00 con 0x7f1ee0078560 2024-04-13T03:30:54.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.776+0000 7f1edaa00640 1 -- 172.21.15.16:0/488373013 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1edc005540 con 0x7f1ee0078560 2024-04-13T03:30:54.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.777+0000 7f1edaa00640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ec80a3bb0 0x7f1ec80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.778+0000 7f1edaa00640 1 -- 172.21.15.16:0/488373013 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1edc0d05d0 con 0x7f1ee0078560 2024-04-13T03:30:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.778+0000 7f1ee5e00640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ec80a3bb0 0x7f1ec80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:54.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.778+0000 7f1ee5e00640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ec80a3bb0 0x7f1ec80a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1ed402c8e0 tx=0x7f1ed402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:54.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:54.782+0000 7f1edaa00640 1 -- 172.21.15.16:0/488373013 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1edc09d3b0 con 0x7f1ee0078560 2024-04-13T03:30:54.896 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6832/3805092939,v1:172.21.15.188:6833/3805092939] up:boot 2024-04-13T03:30:54.896 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: fsmap 1 up:standby 2024-04-13T03:30:54.896 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-13T03:30:54.896 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:54.896 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:54.897 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:54.897 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:54.897 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:30:54.897 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:30:54.897 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:54.897 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:54 smithi119 ceph-mon[35835]: Deploying daemon mds.k on smithi119 2024-04-13T03:30:55.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.031+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ee0062bc0 con 0x7f1ee0078560 2024-04-13T03:30:55.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.032+0000 7f1edaa00640 1 -- 172.21.15.16:0/488373013 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f1edc09ccb0 con 0x7f1ee0078560 2024-04-13T03:30:55.034 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:30:55.034 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-13T03:30:55.034 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-04-13T03:30:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ec80a3bb0 msgr2=0x7f1ec80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ec80a3bb0 0x7f1ec80a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1ed402c8e0 tx=0x7f1ed402df90 comp rx=0 tx=0).stop 2024-04-13T03:30:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ee0078560 msgr2=0x7f1ee01a5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ee0078560 0x7f1ee01a5cc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009fd0 tx=0x7f1edc00ef10 comp rx=0 tx=0).stop 2024-04-13T03:30:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 shutdown_connections 2024-04-13T03:30:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ec80a3bb0 0x7f1ec80a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ee0078560 0x7f1ee01a5cc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ee0077bc0 0x7f1ee019ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 --2- 172.21.15.16:0/488373013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ee0076980 0x7f1ee019e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 >> 172.21.15.16:0/488373013 conn(0x7f1ee00ff5d0 msgr2=0x7f1ee0100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 shutdown_connections 2024-04-13T03:30:55.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:55.035+0000 7f1ee7200640 1 -- 172.21.15.16:0/488373013 wait complete. 2024-04-13T03:30:55.053 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6832/3805092939,v1:172.21.15.188:6833/3805092939] up:boot 2024-04-13T03:30:55.053 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: fsmap 1 up:standby 2024-04-13T03:30:55.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-13T03:30:55.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:55.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:55.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:55.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:55.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:30:55.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:30:55.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:55.054 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:54 smithi016 ceph-mon[31772]: Deploying daemon mds.k on smithi119 2024-04-13T03:30:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6832/3805092939,v1:172.21.15.188:6833/3805092939] up:boot 2024-04-13T03:30:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: fsmap 1 up:standby 2024-04-13T03:30:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-13T03:30:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:30:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:30:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:55.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:54 smithi188 ceph-mon[36398]: Deploying daemon mds.k on smithi119 2024-04-13T03:30:56.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:55 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/488373013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:30:56.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:55 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/488373013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:30:56.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:55 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/488373013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:30:56.521 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:30:56 smithi119 systemd[1]: Starting Ceph mds.k for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:30:56.792 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:30:56 smithi119 podman[65277]: 2024-04-13 03:30:56.521112567 +0000 UTC m=+0.110790495 container create 4e18168fa20f0af1c16cfe2f81fd15cfdfd4a9beaa70c4e013a83e9d84fd331f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-04-13T03:30:56.792 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:30:56 smithi119 podman[65277]: 2024-04-13 03:30:56.427422402 +0000 UTC m=+0.017100335 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:30:56.792 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:30:56 smithi119 podman[65277]: 2024-04-13 03:30:56.76575289 +0000 UTC m=+0.355430798 container init 4e18168fa20f0af1c16cfe2f81fd15cfdfd4a9beaa70c4e013a83e9d84fd331f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:30:56.792 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:30:56 smithi119 podman[65277]: 2024-04-13 03:30:56.768596829 +0000 UTC m=+0.358274730 container start 4e18168fa20f0af1c16cfe2f81fd15cfdfd4a9beaa70c4e013a83e9d84fd331f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:30:57.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:30:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:56 smithi119 ceph-mon[35835]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:57.071 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:30:56 smithi119 ceph-mds[65296]: -- 172.21.15.119:0/3151851480 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d446cfa40 con 0x562d446a3000 2024-04-13T03:30:57.071 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:30:56 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k[65292]: starting mds.k at 2024-04-13T03:30:57.071 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:30:56 smithi119 bash[65277]: 4e18168fa20f0af1c16cfe2f81fd15cfdfd4a9beaa70c4e013a83e9d84fd331f 2024-04-13T03:30:57.071 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:30:56 smithi119 systemd[1]: Started Ceph mds.k for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:30:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:56 smithi016 ceph-mon[31772]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:56 smithi188 ceph-mon[36398]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.264+0000 7f759cc00640 1 -- 172.21.15.16:0/334691825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598101f30 msgr2=0x7f75981023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.264+0000 7f759cc00640 1 --2- 172.21.15.16:0/334691825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598101f30 0x7f75981023b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7580009900 tx=0x7f758002bc30 comp rx=0 tx=0).stop 2024-04-13T03:30:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.264+0000 7f759cc00640 1 -- 172.21.15.16:0/334691825 shutdown_connections 2024-04-13T03:30:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.264+0000 7f759cc00640 1 --2- 172.21.15.16:0/334691825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75981028f0 0x7f7598109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.264+0000 7f759cc00640 1 --2- 172.21.15.16:0/334691825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7598101f30 0x7f75981023b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.264+0000 7f759cc00640 1 --2- 172.21.15.16:0/334691825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7598100d40 0x7f7598101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.264+0000 7f759cc00640 1 -- 172.21.15.16:0/334691825 >> 172.21.15.16:0/334691825 conn(0x7f75980fc290 msgr2=0x7f75980fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.264+0000 7f759cc00640 1 -- 172.21.15.16:0/334691825 shutdown_connections 2024-04-13T03:30:57.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.264+0000 7f759cc00640 1 -- 172.21.15.16:0/334691825 wait complete. 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.265+0000 7f759cc00640 1 Processor -- start 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.265+0000 7f759cc00640 1 -- start start 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.265+0000 7f759cc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7598100d40 0x7f759819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.265+0000 7f759cc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7598101f30 0x7f759819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.265+0000 7f759cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75981028f0 0x7f75981a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.265+0000 7f759cc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f759810c9a0 con 0x7f75981028f0 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.265+0000 7f759cc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f759810c820 con 0x7f7598100d40 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.265+0000 7f759cc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f759810cb20 con 0x7f7598101f30 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75981028f0 0x7f75981a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75981028f0 0x7f75981a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36240/0 (socket says 172.21.15.16:36240) 2024-04-13T03:30:57.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597e00640 1 -- 172.21.15.16:0/949830933 learned_addr learned my addr 172.21.15.16:0/949830933 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597400640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7598100d40 0x7f759819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f758fe00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7598101f30 0x7f759819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597400640 1 -- 172.21.15.16:0/949830933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7598101f30 msgr2=0x7f759819a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597400640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7598101f30 0x7f759819a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597400640 1 -- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75981028f0 msgr2=0x7f75981a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597400640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75981028f0 0x7f75981a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597400640 1 -- 172.21.15.16:0/949830933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75981a1f60 con 0x7f7598100d40 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597e00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75981028f0 0x7f75981a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.266+0000 7f7597400640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7598100d40 0x7f759819a340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f758800e9c0 tx=0x7f758800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.267+0000 7f7594c00640 1 -- 172.21.15.16:0/949830933 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758800cd80 con 0x7f7598100d40 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.267+0000 7f7594c00640 1 -- 172.21.15.16:0/949830933 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7588010910 con 0x7f7598100d40 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.267+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75981a2250 con 0x7f7598100d40 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.267+0000 7f7594c00640 1 -- 172.21.15.16:0/949830933 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588018980 con 0x7f7598100d40 2024-04-13T03:30:57.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.267+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75981a27c0 con 0x7f7598100d40 2024-04-13T03:30:57.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.269+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7598068d00 con 0x7f7598100d40 2024-04-13T03:30:57.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.270+0000 7f7594c00640 1 -- 172.21.15.16:0/949830933 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f758801f020 con 0x7f7598100d40 2024-04-13T03:30:57.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.271+0000 7f7594c00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f757c0a3c30 0x7f757c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.272+0000 7f7594c00640 1 -- 172.21.15.16:0/949830933 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f75880cff10 con 0x7f7598100d40 2024-04-13T03:30:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.272+0000 7f758fe00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f757c0a3c30 0x7f757c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:57.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.273+0000 7f758fe00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f757c0a3c30 0x7f757c0a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f758002c7b0 tx=0x7f7580005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:57.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.275+0000 7f7594c00640 1 -- 172.21.15.16:0/949830933 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f758809ccf0 con 0x7f7598100d40 2024-04-13T03:30:57.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.531+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75981a2a20 con 0x7f7598100d40 2024-04-13T03:30:57.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.531+0000 7f7594c00640 1 -- 172.21.15.16:0/949830933 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f758809c5f0 con 0x7f7598100d40 2024-04-13T03:30:57.533 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:30:57.533 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-13T03:30:57.533 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.534+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f757c0a3c30 msgr2=0x7f757c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.534+0000 7f759cc00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f757c0a3c30 0x7f757c0a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f758002c7b0 tx=0x7f7580005fd0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.534+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7598100d40 msgr2=0x7f759819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.534+0000 7f759cc00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7598100d40 0x7f759819a340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f758800e9c0 tx=0x7f758800ee80 comp rx=0 tx=0).stop 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.534+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 shutdown_connections 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.535+0000 7f759cc00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f757c0a3c30 0x7f757c0a60f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.535+0000 7f759cc00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75981028f0 0x7f75981a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.535+0000 7f759cc00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7598101f30 0x7f759819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.535+0000 7f759cc00640 1 --2- 172.21.15.16:0/949830933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7598100d40 0x7f759819a340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.535+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 >> 172.21.15.16:0/949830933 conn(0x7f75980fc290 msgr2=0x7f75980fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.535+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 shutdown_connections 2024-04-13T03:30:57.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:57.535+0000 7f759cc00640 1 -- 172.21.15.16:0/949830933 wait complete. 2024-04-13T03:30:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6832/3221738952,v1:172.21.15.119:6833/3221738952] up:boot 2024-04-13T03:30:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: fsmap 2 up:standby 2024-04-13T03:30:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-13T03:30:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:30:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:30:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:58.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: Deploying daemon mds.j on smithi016 2024-04-13T03:30:58.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:57 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/949830933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:30:58.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6832/3221738952,v1:172.21.15.119:6833/3221738952] up:boot 2024-04-13T03:30:58.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: fsmap 2 up:standby 2024-04-13T03:30:58.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-13T03:30:58.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:58.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:58.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:58.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:30:58.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:30:58.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:58.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: Deploying daemon mds.j on smithi016 2024-04-13T03:30:58.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:57 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/949830933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:30:58.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6832/3221738952,v1:172.21.15.119:6833/3221738952] up:boot 2024-04-13T03:30:58.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: fsmap 2 up:standby 2024-04-13T03:30:58.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-13T03:30:58.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:58.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:58.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:30:58.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:30:58.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:30:58.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:58.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: Deploying daemon mds.j on smithi016 2024-04-13T03:30:58.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:57 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/949830933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:30:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:30:58 smithi188 ceph-mon[36398]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:59.186 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:30:58 smithi119 ceph-mon[35835]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:59.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:30:58 smithi016 ceph-mon[31772]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:59.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:30:59.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.883+0000 7fe666200640 1 -- 172.21.15.16:0/3836023831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66010e6d0 msgr2=0x7fe660110b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.883+0000 7fe666200640 1 --2- 172.21.15.16:0/3836023831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66010e6d0 0x7fe660110b90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00c5d0 tx=0x7fe64c02ba30 comp rx=0 tx=0).stop 2024-04-13T03:30:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.883+0000 7fe666200640 1 -- 172.21.15.16:0/3836023831 shutdown_connections 2024-04-13T03:30:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.883+0000 7fe666200640 1 --2- 172.21.15.16:0/3836023831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66010e6d0 0x7fe660110b90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.883+0000 7fe666200640 1 --2- 172.21.15.16:0/3836023831 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe66010bbf0 0x7fe66010e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.883+0000 7fe666200640 1 --2- 172.21.15.16:0/3836023831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe660109230 0x7fe66010b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.883+0000 7fe666200640 1 -- 172.21.15.16:0/3836023831 >> 172.21.15.16:0/3836023831 conn(0x7fe66006ee60 msgr2=0x7fe6600712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:30:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.883+0000 7fe666200640 1 -- 172.21.15.16:0/3836023831 shutdown_connections 2024-04-13T03:30:59.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.883+0000 7fe666200640 1 -- 172.21.15.16:0/3836023831 wait complete. 2024-04-13T03:30:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.884+0000 7fe666200640 1 Processor -- start 2024-04-13T03:30:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.885+0000 7fe666200640 1 -- start start 2024-04-13T03:30:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.885+0000 7fe666200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660109230 0x7fe6601a71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.885+0000 7fe666200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66010bbf0 0x7fe6601a7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:59.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.885+0000 7fe666200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe66010e6d0 0x7fe6601ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:59.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.885+0000 7fe666200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe660113950 con 0x7fe66010bbf0 2024-04-13T03:30:59.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.885+0000 7fe666200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe6601137d0 con 0x7fe66010e6d0 2024-04-13T03:30:59.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.885+0000 7fe666200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe660113ad0 con 0x7fe660109230 2024-04-13T03:30:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.886+0000 7fe65f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660109230 0x7fe6601a71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.886+0000 7fe65f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660109230 0x7fe6601a71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:36446/0 (socket says 172.21.15.16:36446) 2024-04-13T03:30:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.886+0000 7fe65f400640 1 -- 172.21.15.16:0/2504618614 learned_addr learned my addr 172.21.15.16:0/2504618614 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:30:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.886+0000 7fe65ea00640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66010bbf0 0x7fe6601a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.886+0000 7fe65fe00640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe66010e6d0 0x7fe6601ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.886+0000 7fe65fe00640 1 -- 172.21.15.16:0/2504618614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660109230 msgr2=0x7fe6601a71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.886+0000 7fe65fe00640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660109230 0x7fe6601a71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:59.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.887+0000 7fe65fe00640 1 -- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66010bbf0 msgr2=0x7fe6601a7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:30:59.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.887+0000 7fe65fe00640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66010bbf0 0x7fe6601a7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:30:59.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.887+0000 7fe65fe00640 1 -- 172.21.15.16:0/2504618614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6601aedf0 con 0x7fe66010e6d0 2024-04-13T03:30:59.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.887+0000 7fe65ea00640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66010bbf0 0x7fe6601a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:30:59.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.887+0000 7fe65fe00640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe66010e6d0 0x7fe6601ae780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00c5a0 tx=0x7fe64c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:59.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.887+0000 7fe657e00640 1 -- 172.21.15.16:0/2504618614 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c033070 con 0x7fe66010e6d0 2024-04-13T03:30:59.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.887+0000 7fe657e00640 1 -- 172.21.15.16:0/2504618614 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe64c0054b0 con 0x7fe66010e6d0 2024-04-13T03:30:59.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.887+0000 7fe657e00640 1 -- 172.21.15.16:0/2504618614 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c00a0d0 con 0x7fe66010e6d0 2024-04-13T03:30:59.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.887+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6601af080 con 0x7fe66010e6d0 2024-04-13T03:30:59.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.888+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6601af550 con 0x7fe66010e6d0 2024-04-13T03:30:59.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.890+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe660068de0 con 0x7fe66010e6d0 2024-04-13T03:30:59.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.891+0000 7fe657e00640 1 -- 172.21.15.16:0/2504618614 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe64c004fd0 con 0x7fe66010e6d0 2024-04-13T03:30:59.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.892+0000 7fe657e00640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6440a3c80 0x7fe6440a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:30:59.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.892+0000 7fe657e00640 1 -- 172.21.15.16:0/2504618614 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe64c0f1760 con 0x7fe66010e6d0 2024-04-13T03:30:59.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.893+0000 7fe65f400640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6440a3c80 0x7fe6440a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:30:59.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.893+0000 7fe65f400640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6440a3c80 0x7fe6440a6140 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe658005c40 tx=0x7fe65800f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:30:59.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:30:59.896+0000 7fe657e00640 1 -- 172.21.15.16:0/2504618614 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe64c046060 con 0x7fe66010e6d0 2024-04-13T03:31:00.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.152+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6601af870 con 0x7fe66010e6d0 2024-04-13T03:31:00.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.153+0000 7fe657e00640 1 -- 172.21.15.16:0/2504618614 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fe64c0be540 con 0x7fe66010e6d0 2024-04-13T03:31:00.154 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:00.154 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-13T03:31:00.154 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2024-04-13T03:31:00.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6440a3c80 msgr2=0x7fe6440a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6440a3c80 0x7fe6440a6140 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe658005c40 tx=0x7fe65800f040 comp rx=0 tx=0).stop 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe66010e6d0 msgr2=0x7fe6601ae780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe66010e6d0 0x7fe6601ae780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00c5a0 tx=0x7fe64c004270 comp rx=0 tx=0).stop 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 shutdown_connections 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6440a3c80 0x7fe6440a6140 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe66010e6d0 0x7fe6601ae780 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe66010bbf0 0x7fe6601a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 --2- 172.21.15.16:0/2504618614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe660109230 0x7fe6601a71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.155+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 >> 172.21.15.16:0/2504618614 conn(0x7fe66006ee60 msgr2=0x7fe66010dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.156+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 shutdown_connections 2024-04-13T03:31:00.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:00.156+0000 7fe666200640 1 -- 172.21.15.16:0/2504618614 wait complete. 2024-04-13T03:31:00.777 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:31:00 smithi016 podman[69175]: 2024-04-13 03:31:00.529598615 +0000 UTC m=+0.192778014 container create 1495192ece6d0a588285f70dd8680aec62b9c0eb4464e2a85f533169a9e7d488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-13T03:31:00.777 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:31:00 smithi016 podman[69175]: 2024-04-13 03:31:00.775207578 +0000 UTC m=+0.438387010 container init 1495192ece6d0a588285f70dd8680aec62b9c0eb4464e2a85f533169a9e7d488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True) 2024-04-13T03:31:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:00 smithi016 ceph-mon[31772]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:00 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2504618614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:01.105 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:31:00 smithi016 podman[69175]: 2024-04-13 03:31:00.784473089 +0000 UTC m=+0.447652462 container start 1495192ece6d0a588285f70dd8680aec62b9c0eb4464e2a85f533169a9e7d488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:31:01.105 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:31:00 smithi016 ceph-mds[69194]: -- 172.21.15.16:0/894317834 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564fa4135a40 con 0x564fa4108800 2024-04-13T03:31:01.105 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:31:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j[69190]: starting mds.j at 2024-04-13T03:31:01.105 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:31:00 smithi016 bash[69175]: 1495192ece6d0a588285f70dd8680aec62b9c0eb4464e2a85f533169a9e7d488 2024-04-13T03:31:01.105 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:31:00 smithi016 systemd[1]: Started Ceph mds.j for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:00 smithi119 ceph-mon[35835]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:00 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2504618614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:00 smithi188 ceph-mon[36398]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:00 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2504618614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:02.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.16:6834/1177424525,v1:172.21.15.16:6835/1177424525] up:boot 2024-04-13T03:31:02.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: fsmap 3 up:standby 2024-04-13T03:31:02.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-13T03:31:02.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:02.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:02.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:02.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:02.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:02.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:02.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:01 smithi016 ceph-mon[31772]: Deploying daemon mds.d on smithi016 2024-04-13T03:31:02.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.16:6834/1177424525,v1:172.21.15.16:6835/1177424525] up:boot 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: fsmap 3 up:standby 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:02.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:01 smithi119 ceph-mon[35835]: Deploying daemon mds.d on smithi016 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.16:6834/1177424525,v1:172.21.15.16:6835/1177424525] up:boot 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: fsmap 3 up:standby 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:02.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:01 smithi188 ceph-mon[36398]: Deploying daemon mds.d on smithi016 2024-04-13T03:31:02.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.417+0000 7ffab2a00640 1 -- 172.21.15.16:0/287038409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac1019b0 msgr2=0x7ffaac101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:02.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.417+0000 7ffab2a00640 1 --2- 172.21.15.16:0/287038409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac1019b0 0x7ffaac101e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009be0 tx=0x7ffaa402bef0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.417+0000 7ffab2a00640 1 -- 172.21.15.16:0/287038409 shutdown_connections 2024-04-13T03:31:02.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.417+0000 7ffab2a00640 1 --2- 172.21.15.16:0/287038409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac1019b0 0x7ffaac101e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.417+0000 7ffab2a00640 1 --2- 172.21.15.16:0/287038409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaac101010 0x7ffaac101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.417+0000 7ffab2a00640 1 --2- 172.21.15.16:0/287038409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaac1006b0 0x7ffaac100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.417+0000 7ffab2a00640 1 -- 172.21.15.16:0/287038409 >> 172.21.15.16:0/287038409 conn(0x7ffaac0fc2d0 msgr2=0x7ffaac0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:02.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.417+0000 7ffab2a00640 1 -- 172.21.15.16:0/287038409 shutdown_connections 2024-04-13T03:31:02.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.417+0000 7ffab2a00640 1 -- 172.21.15.16:0/287038409 wait complete. 2024-04-13T03:31:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.418+0000 7ffab2a00640 1 Processor -- start 2024-04-13T03:31:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.418+0000 7ffab2a00640 1 -- start start 2024-04-13T03:31:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.418+0000 7ffab2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaac1006b0 0x7ffaac19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.418+0000 7ffab2a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaac101010 0x7ffaac19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.418+0000 7ffab2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac1019b0 0x7ffaac1a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.418+0000 7ffab2a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac10fd30 con 0x7ffaac1006b0 2024-04-13T03:31:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.418+0000 7ffab2a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac10fbb0 con 0x7ffaac1019b0 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.418+0000 7ffab2a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac10feb0 con 0x7ffaac101010 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab1600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaac1006b0 0x7ffaac19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab0c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaac101010 0x7ffaac19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab0c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaac101010 0x7ffaac19a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:36472/0 (socket says 172.21.15.16:36472) 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab0c00640 1 -- 172.21.15.16:0/3777830481 learned_addr learned my addr 172.21.15.16:0/3777830481 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab2000640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac1019b0 0x7ffaac1a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab0c00640 1 -- 172.21.15.16:0/3777830481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac1019b0 msgr2=0x7ffaac1a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab0c00640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac1019b0 0x7ffaac1a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab0c00640 1 -- 172.21.15.16:0/3777830481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaac1006b0 msgr2=0x7ffaac19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab0c00640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaac1006b0 0x7ffaac19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab0c00640 1 -- 172.21.15.16:0/3777830481 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac1a1ef0 con 0x7ffaac101010 2024-04-13T03:31:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.419+0000 7ffab2000640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac1019b0 0x7ffaac1a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:02.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.420+0000 7ffab0c00640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaac101010 0x7ffaac19a810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800e9f0 tx=0x7ffa9800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:02.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.420+0000 7ffaaa000640 1 -- 172.21.15.16:0/3777830481 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800cd80 con 0x7ffaac101010 2024-04-13T03:31:02.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.420+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac1a21e0 con 0x7ffaac101010 2024-04-13T03:31:02.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.420+0000 7ffaaa000640 1 -- 172.21.15.16:0/3777830481 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa98005500 con 0x7ffaac101010 2024-04-13T03:31:02.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.420+0000 7ffaaa000640 1 -- 172.21.15.16:0/3777830481 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98010690 con 0x7ffaac101010 2024-04-13T03:31:02.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.420+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac1a26e0 con 0x7ffaac101010 2024-04-13T03:31:02.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.422+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaac10b510 con 0x7ffaac101010 2024-04-13T03:31:02.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.423+0000 7ffaaa000640 1 -- 172.21.15.16:0/3777830481 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffa980046c0 con 0x7ffaac101010 2024-04-13T03:31:02.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.425+0000 7ffaaa000640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ffa940a3bb0 0x7ffa940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:02.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.425+0000 7ffaaa000640 1 -- 172.21.15.16:0/3777830481 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ffa980d0400 con 0x7ffaac101010 2024-04-13T03:31:02.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.425+0000 7ffab1600640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ffa940a3bb0 0x7ffa940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:02.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.426+0000 7ffab1600640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ffa940a3bb0 0x7ffa940a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009530 tx=0x7ffaa0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:02.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.429+0000 7ffaaa000640 1 -- 172.21.15.16:0/3777830481 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa9809d1e0 con 0x7ffaac101010 2024-04-13T03:31:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.685+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffaac02d5e0 con 0x7ffaac101010 2024-04-13T03:31:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.686+0000 7ffaaa000640 1 -- 172.21.15.16:0/3777830481 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7ffa9809cae0 con 0x7ffaac101010 2024-04-13T03:31:02.688 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:02.688 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-04-13T03:31:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 4 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ffa940a3bb0 msgr2=0x7ffa940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ffa940a3bb0 0x7ffa940a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009530 tx=0x7ffaa0009260 comp rx=0 tx=0).stop 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaac101010 msgr2=0x7ffaac19a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaac101010 0x7ffaac19a810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800e9f0 tx=0x7ffa9800edf0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 shutdown_connections 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ffa940a3bb0 0x7ffa940a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac1019b0 0x7ffaac1a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaac101010 0x7ffaac19a810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 --2- 172.21.15.16:0/3777830481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffaac1006b0 0x7ffaac19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.690+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 >> 172.21.15.16:0/3777830481 conn(0x7ffaac0fc2d0 msgr2=0x7ffaac0fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.691+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 shutdown_connections 2024-04-13T03:31:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:02.691+0000 7ffab2a00640 1 -- 172.21.15.16:0/3777830481 wait complete. 2024-04-13T03:31:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:03 smithi188 ceph-mon[36398]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:03.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:03 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3777830481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:03.499 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:03 smithi016 ceph-mon[31772]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:03.499 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:03 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3777830481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:03.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:03 smithi119 ceph-mon[35835]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:03.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:03 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3777830481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:04.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:04.767 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:31:04 smithi016 systemd[1]: Starting Ceph mds.d for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:31:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.939+0000 7fb41f200640 1 -- 172.21.15.16:0/3665709650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4201051f0 msgr2=0x7fb4201075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.939+0000 7fb41f200640 1 --2- 172.21.15.16:0/3665709650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4201051f0 0x7fb4201075e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009860 tx=0x7fb40c02bb70 comp rx=0 tx=0).stop 2024-04-13T03:31:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.939+0000 7fb41f200640 1 -- 172.21.15.16:0/3665709650 shutdown_connections 2024-04-13T03:31:04.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.939+0000 7fb41f200640 1 --2- 172.21.15.16:0/3665709650 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb420107b20 0x7fb420109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.939+0000 7fb41f200640 1 --2- 172.21.15.16:0/3665709650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4201051f0 0x7fb4201075e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.939+0000 7fb41f200640 1 --2- 172.21.15.16:0/3665709650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4201028c0 0x7fb420104cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.939+0000 7fb41f200640 1 -- 172.21.15.16:0/3665709650 >> 172.21.15.16:0/3665709650 conn(0x7fb4200fc290 msgr2=0x7fb4200fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.939+0000 7fb41f200640 1 -- 172.21.15.16:0/3665709650 shutdown_connections 2024-04-13T03:31:04.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.939+0000 7fb41f200640 1 -- 172.21.15.16:0/3665709650 wait complete. 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.940+0000 7fb41f200640 1 Processor -- start 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41f200640 1 -- start start 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4201028c0 0x7fb42019a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4201051f0 0x7fb42019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41f200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb420107b20 0x7fb4201a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb42010dab0 con 0x7fb4201028c0 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41f200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb42010d930 con 0x7fb420107b20 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb42010dc30 con 0x7fb4201051f0 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4201028c0 0x7fb42019a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4201051f0 0x7fb42019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:04.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4201028c0 0x7fb42019a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37558/0 (socket says 172.21.15.16:37558) 2024-04-13T03:31:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41de00640 1 -- 172.21.15.16:0/1300600411 learned_addr learned my addr 172.21.15.16:0/1300600411 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4201051f0 0x7fb42019a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:36486/0 (socket says 172.21.15.16:36486) 2024-04-13T03:31:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.941+0000 7fb41e800640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb420107b20 0x7fb4201a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb41d400640 1 -- 172.21.15.16:0/1300600411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb420107b20 msgr2=0x7fb4201a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb41d400640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb420107b20 0x7fb4201a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb41d400640 1 -- 172.21.15.16:0/1300600411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4201028c0 msgr2=0x7fb42019a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb41d400640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4201028c0 0x7fb42019a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb41d400640 1 -- 172.21.15.16:0/1300600411 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4201a1fe0 con 0x7fb4201051f0 2024-04-13T03:31:04.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb41d400640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4201051f0 0x7fb42019a900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb40c02c0b0 tx=0x7fb40c032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb416a00640 1 -- 172.21.15.16:0/1300600411 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c041020 con 0x7fb4201051f0 2024-04-13T03:31:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4201a2270 con 0x7fb4201051f0 2024-04-13T03:31:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb416a00640 1 -- 172.21.15.16:0/1300600411 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb40c02c9f0 con 0x7fb4201051f0 2024-04-13T03:31:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.942+0000 7fb416a00640 1 -- 172.21.15.16:0/1300600411 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c039040 con 0x7fb4201051f0 2024-04-13T03:31:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.943+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4201a27b0 con 0x7fb4201051f0 2024-04-13T03:31:04.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.944+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb420109670 con 0x7fb4201051f0 2024-04-13T03:31:04.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.946+0000 7fb416a00640 1 -- 172.21.15.16:0/1300600411 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb40c044020 con 0x7fb4201051f0 2024-04-13T03:31:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.947+0000 7fb416a00640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4040a3bb0 0x7fb4040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.947+0000 7fb416a00640 1 -- 172.21.15.16:0/1300600411 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb40c0f1b60 con 0x7fb4201051f0 2024-04-13T03:31:04.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.948+0000 7fb41de00640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4040a3bb0 0x7fb4040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:04.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.948+0000 7fb41de00640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4040a3bb0 0x7fb4040a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb4100095f0 tx=0x7fb410005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:04.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:04.950+0000 7fb416a00640 1 -- 172.21.15.16:0/1300600411 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb40c04b020 con 0x7fb4201051f0 2024-04-13T03:31:05.050 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:31:04 smithi016 podman[69522]: 2024-04-13 03:31:04.673630323 +0000 UTC m=+0.036452322 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:31:05.050 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:31:04 smithi016 podman[69522]: 2024-04-13 03:31:04.784295451 +0000 UTC m=+0.147117437 container create 050a364f12310c57d2dc308c1293a3eecbc1476f4493c933f8ec04a10c655c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:31:05.050 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:31:05 smithi016 podman[69522]: 2024-04-13 03:31:05.046945121 +0000 UTC m=+0.409767177 container init 050a364f12310c57d2dc308c1293a3eecbc1476f4493c933f8ec04a10c655c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-13T03:31:05.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.211+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb42002d5e0 con 0x7fb4201051f0 2024-04-13T03:31:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.212+0000 7fb416a00640 1 -- 172.21.15.16:0/1300600411 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7fb40c0be9c0 con 0x7fb4201051f0 2024-04-13T03:31:05.213 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:05.213 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-13T03:31:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 5 2024-04-13T03:31:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4040a3bb0 msgr2=0x7fb4040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4040a3bb0 0x7fb4040a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb4100095f0 tx=0x7fb410005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4201051f0 msgr2=0x7fb42019a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4201051f0 0x7fb42019a900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb40c02c0b0 tx=0x7fb40c032430 comp rx=0 tx=0).stop 2024-04-13T03:31:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 shutdown_connections 2024-04-13T03:31:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4040a3bb0 0x7fb4040a6070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb420107b20 0x7fb4201a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4201051f0 0x7fb42019a900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 --2- 172.21.15.16:0/1300600411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4201028c0 0x7fb42019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 >> 172.21.15.16:0/1300600411 conn(0x7fb4200fc290 msgr2=0x7fb420102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 shutdown_connections 2024-04-13T03:31:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:05.215+0000 7fb41f200640 1 -- 172.21.15.16:0/1300600411 wait complete. 2024-04-13T03:31:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:05 smithi016 ceph-mon[31772]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:05.355 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:31:05 smithi016 podman[69522]: 2024-04-13 03:31:05.057303976 +0000 UTC m=+0.420125956 container start 050a364f12310c57d2dc308c1293a3eecbc1476f4493c933f8ec04a10c655c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-13T03:31:05.355 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:31:05 smithi016 ceph-mds[69580]: -- 172.21.15.16:0/1691915335 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556e67c39a40 con 0x556e67c0c800 2024-04-13T03:31:05.355 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:31:05 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d[69557]: starting mds.d at 2024-04-13T03:31:05.355 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:31:05 smithi016 bash[69522]: 050a364f12310c57d2dc308c1293a3eecbc1476f4493c933f8ec04a10c655c2c 2024-04-13T03:31:05.355 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:31:05 smithi016 systemd[1]: Started Ceph mds.d for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:05.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:05 smithi119 ceph-mon[35835]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:05.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:05 smithi188 ceph-mon[36398]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:06.261 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.16:6836/2782624805,v1:172.21.15.16:6837/2782624805] up:boot 2024-04-13T03:31:06.261 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: fsmap 4 up:standby 2024-04-13T03:31:06.261 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-13T03:31:06.261 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1300600411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:06.261 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:06.261 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:06.261 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:06.261 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:06.261 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:06.262 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:06.262 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:06 smithi119 ceph-mon[35835]: Deploying daemon mds.e on smithi119 2024-04-13T03:31:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.16:6836/2782624805,v1:172.21.15.16:6837/2782624805] up:boot 2024-04-13T03:31:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: fsmap 4 up:standby 2024-04-13T03:31:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-13T03:31:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1300600411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:06.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:06.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:06.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:06 smithi188 ceph-mon[36398]: Deploying daemon mds.e on smithi119 2024-04-13T03:31:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.16:6836/2782624805,v1:172.21.15.16:6837/2782624805] up:boot 2024-04-13T03:31:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: fsmap 4 up:standby 2024-04-13T03:31:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-13T03:31:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1300600411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:06 smithi016 ceph-mon[31772]: Deploying daemon mds.e on smithi119 2024-04-13T03:31:07.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:07.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.473+0000 7f54ba600640 1 -- 172.21.15.16:0/1165211107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b4101070 msgr2=0x7f54b41014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:07.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.473+0000 7f54ba600640 1 --2- 172.21.15.16:0/1165211107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b4101070 0x7f54b41014d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009900 tx=0x7f54a002bc50 comp rx=0 tx=0).stop 2024-04-13T03:31:07.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.474+0000 7f54ba600640 1 -- 172.21.15.16:0/1165211107 shutdown_connections 2024-04-13T03:31:07.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.474+0000 7f54ba600640 1 --2- 172.21.15.16:0/1165211107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54b4101a10 0x7f54b4101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.474+0000 7f54ba600640 1 --2- 172.21.15.16:0/1165211107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b4101070 0x7f54b41014d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.474+0000 7f54ba600640 1 --2- 172.21.15.16:0/1165211107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4100710 0x7f54b4100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.474+0000 7f54ba600640 1 -- 172.21.15.16:0/1165211107 >> 172.21.15.16:0/1165211107 conn(0x7f54b40fc290 msgr2=0x7f54b40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.474+0000 7f54ba600640 1 -- 172.21.15.16:0/1165211107 shutdown_connections 2024-04-13T03:31:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.474+0000 7f54ba600640 1 -- 172.21.15.16:0/1165211107 wait complete. 2024-04-13T03:31:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.475+0000 7f54ba600640 1 Processor -- start 2024-04-13T03:31:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.475+0000 7f54ba600640 1 -- start start 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54ba600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4100710 0x7f54b419a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54ba600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b4101070 0x7f54b419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54ba600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54b4101a10 0x7f54b41a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54ba600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f54b410fcf0 con 0x7f54b4100710 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54ba600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f54b410fb70 con 0x7f54b4101070 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54ba600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f54b410fe70 con 0x7f54b4101a10 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b9200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4100710 0x7f54b419a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b9200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4100710 0x7f54b419a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37606/0 (socket says 172.21.15.16:37606) 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b9200640 1 -- 172.21.15.16:0/2998605299 learned_addr learned my addr 172.21.15.16:0/2998605299 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b3e00640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b4101070 0x7f54b419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b9c00640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54b4101a10 0x7f54b41a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b9200640 1 -- 172.21.15.16:0/2998605299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54b4101a10 msgr2=0x7f54b41a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b9200640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54b4101a10 0x7f54b41a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b9200640 1 -- 172.21.15.16:0/2998605299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b4101070 msgr2=0x7f54b419a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b9200640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b4101070 0x7f54b419a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.476+0000 7f54b9200640 1 -- 172.21.15.16:0/2998605299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b41a1f80 con 0x7f54b4100710 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.477+0000 7f54b9200640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4100710 0x7f54b419a3d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e9f0 tx=0x7f54a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.477+0000 7f54b1600640 1 -- 172.21.15.16:0/2998605299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a800cd80 con 0x7f54b4100710 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.477+0000 7f54b1600640 1 -- 172.21.15.16:0/2998605299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a8010910 con 0x7f54b4100710 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.477+0000 7f54b1600640 1 -- 172.21.15.16:0/2998605299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8018990 con 0x7f54b4100710 2024-04-13T03:31:07.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.477+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b41a2270 con 0x7f54b4100710 2024-04-13T03:31:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.477+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b41a2790 con 0x7f54b4100710 2024-04-13T03:31:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.478+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54b410b460 con 0x7f54b4100710 2024-04-13T03:31:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.481+0000 7f54b1600640 1 -- 172.21.15.16:0/2998605299 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f54a8010ac0 con 0x7f54b4100710 2024-04-13T03:31:07.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.482+0000 7f54b1600640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f549c0a3bb0 0x7f549c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:07.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.483+0000 7f54b1600640 1 -- 172.21.15.16:0/2998605299 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f54a80d0730 con 0x7f54b4100710 2024-04-13T03:31:07.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.483+0000 7f54b3e00640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f549c0a3bb0 0x7f549c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:07.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.484+0000 7f54b3e00640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f549c0a3bb0 0x7f549c0a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54a002c260 tx=0x7f54a0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:07.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.485+0000 7f54b1600640 1 -- 172.21.15.16:0/2998605299 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f54a809cec0 con 0x7f54b4100710 2024-04-13T03:31:07.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:07 smithi119 ceph-mon[35835]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:07.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:07 smithi016 ceph-mon[31772]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:07.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:07 smithi188 ceph-mon[36398]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.736+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54b402d5e0 con 0x7f54b4100710 2024-04-13T03:31:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.737+0000 7f54b1600640 1 -- 172.21.15.16:0/2998605299 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f54a80a11c0 con 0x7f54b4100710 2024-04-13T03:31:07.739 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:07.739 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-13T03:31:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 5 2024-04-13T03:31:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f549c0a3bb0 msgr2=0x7f549c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f549c0a3bb0 0x7f549c0a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54a002c260 tx=0x7f54a0000bc0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4100710 msgr2=0x7f54b419a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4100710 0x7f54b419a3d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e9f0 tx=0x7f54a800eeb0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 shutdown_connections 2024-04-13T03:31:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f549c0a3bb0 0x7f549c0a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54b4101a10 0x7f54b41a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b4101070 0x7f54b419a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 --2- 172.21.15.16:0/2998605299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54b4100710 0x7f54b419a3d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 >> 172.21.15.16:0/2998605299 conn(0x7f54b40fc290 msgr2=0x7f54b40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.740+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 shutdown_connections 2024-04-13T03:31:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:07.741+0000 7f54ba600640 1 -- 172.21.15.16:0/2998605299 wait complete. 2024-04-13T03:31:08.321 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:31:08 smithi119 systemd[1]: Started Ceph mds.e for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:08.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:08 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2998605299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:08.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:08 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:08.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:08 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2998605299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:08.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:08.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:08.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:08.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:08.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:08.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:08 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:08 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2998605299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:08.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:08 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:09 smithi188 ceph-mon[36398]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:09.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:09 smithi188 ceph-mon[36398]: Deploying daemon mds.f on smithi188 2024-04-13T03:31:09.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:09 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] up:boot 2024-04-13T03:31:09.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:09 smithi188 ceph-mon[36398]: fsmap 5 up:standby 2024-04-13T03:31:09.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:09 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-13T03:31:09.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:09 smithi016 ceph-mon[31772]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:09 smithi016 ceph-mon[31772]: Deploying daemon mds.f on smithi188 2024-04-13T03:31:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:09 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] up:boot 2024-04-13T03:31:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:09 smithi016 ceph-mon[31772]: fsmap 5 up:standby 2024-04-13T03:31:09.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:09 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-13T03:31:09.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:09 smithi119 ceph-mon[35835]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:09.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:09 smithi119 ceph-mon[35835]: Deploying daemon mds.f on smithi188 2024-04-13T03:31:09.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:09 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] up:boot 2024-04-13T03:31:09.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:09 smithi119 ceph-mon[35835]: fsmap 5 up:standby 2024-04-13T03:31:09.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:09 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-13T03:31:09.763 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.989+0000 7efe59a00640 1 -- 172.21.15.16:0/324454679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54109810 msgr2=0x7efe54109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.989+0000 7efe59a00640 1 --2- 172.21.15.16:0/324454679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54109810 0x7efe54109c90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efe40009900 tx=0x7efe4002bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.989+0000 7efe59a00640 1 -- 172.21.15.16:0/324454679 shutdown_connections 2024-04-13T03:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.989+0000 7efe59a00640 1 --2- 172.21.15.16:0/324454679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe5410a1d0 0x7efe54110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.989+0000 7efe59a00640 1 --2- 172.21.15.16:0/324454679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54109810 0x7efe54109c90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.989+0000 7efe59a00640 1 --2- 172.21.15.16:0/324454679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe54108620 0x7efe54108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.989+0000 7efe59a00640 1 -- 172.21.15.16:0/324454679 >> 172.21.15.16:0/324454679 conn(0x7efe540775e0 msgr2=0x7efe54079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.989+0000 7efe59a00640 1 -- 172.21.15.16:0/324454679 shutdown_connections 2024-04-13T03:31:09.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.989+0000 7efe59a00640 1 -- 172.21.15.16:0/324454679 wait complete. 2024-04-13T03:31:09.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.990+0000 7efe59a00640 1 Processor -- start 2024-04-13T03:31:09.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe59a00640 1 -- start start 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe59a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54108620 0x7efe5419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe59a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe54109810 0x7efe5419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe59a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe5410a1d0 0x7efe541a5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe59a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efe541142b0 con 0x7efe54108620 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe59a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efe54114130 con 0x7efe5410a1d0 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe59a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efe54114430 con 0x7efe54109810 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe52a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54108620 0x7efe5419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe52a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54108620 0x7efe5419e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60850/0 (socket says 172.21.15.16:60850) 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe52a00640 1 -- 172.21.15.16:0/3583312222 learned_addr learned my addr 172.21.15.16:0/3583312222 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe52000640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe54109810 0x7efe5419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.991+0000 7efe53400640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe5410a1d0 0x7efe541a5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.992+0000 7efe52a00640 1 -- 172.21.15.16:0/3583312222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe54109810 msgr2=0x7efe5419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.992+0000 7efe52a00640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe54109810 0x7efe5419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.992+0000 7efe52a00640 1 -- 172.21.15.16:0/3583312222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe5410a1d0 msgr2=0x7efe541a5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.992+0000 7efe52a00640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe5410a1d0 0x7efe541a5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.992+0000 7efe52a00640 1 -- 172.21.15.16:0/3583312222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe541a65a0 con 0x7efe54108620 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.992+0000 7efe52000640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe54109810 0x7efe5419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.992+0000 7efe52a00640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54108620 0x7efe5419e980 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efe4400e9c0 tx=0x7efe4400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.992+0000 7efe4b400640 1 -- 172.21.15.16:0/3583312222 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4400cd80 con 0x7efe54108620 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.992+0000 7efe4b400640 1 -- 172.21.15.16:0/3583312222 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe440035f0 con 0x7efe54108620 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.993+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe541a6890 con 0x7efe54108620 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.993+0000 7efe4b400640 1 -- 172.21.15.16:0/3583312222 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe44010690 con 0x7efe54108620 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.993+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe541a6d60 con 0x7efe54108620 2024-04-13T03:31:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.994+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe5410f990 con 0x7efe54108620 2024-04-13T03:31:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.997+0000 7efe4b400640 1 -- 172.21.15.16:0/3583312222 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efe440037a0 con 0x7efe54108620 2024-04-13T03:31:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.997+0000 7efe4b400640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe380a3bb0 0x7efe380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.997+0000 7efe4b400640 1 -- 172.21.15.16:0/3583312222 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7efe440d0360 con 0x7efe54108620 2024-04-13T03:31:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.998+0000 7efe52000640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe380a3bb0 0x7efe380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:09.998+0000 7efe52000640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe380a3bb0 0x7efe380a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efe4002c7b0 tx=0x7efe40005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:10.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.002+0000 7efe4b400640 1 -- 172.21.15.16:0/3583312222 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe4409d140 con 0x7efe54108620 2024-04-13T03:31:10.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.250+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe5402d5e0 con 0x7efe54108620 2024-04-13T03:31:10.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.252+0000 7efe4b400640 1 -- 172.21.15.16:0/3583312222 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7efe4409ca40 con 0x7efe54108620 2024-04-13T03:31:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:10.253 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-13T03:31:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2024-04-13T03:31:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe380a3bb0 msgr2=0x7efe380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe380a3bb0 0x7efe380a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efe4002c7b0 tx=0x7efe40005fd0 comp rx=0 tx=0).stop 2024-04-13T03:31:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54108620 msgr2=0x7efe5419e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54108620 0x7efe5419e980 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efe4400e9c0 tx=0x7efe4400ee80 comp rx=0 tx=0).stop 2024-04-13T03:31:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 shutdown_connections 2024-04-13T03:31:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efe380a3bb0 0x7efe380a6070 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe5410a1d0 0x7efe541a5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:10.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efe54109810 0x7efe5419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:10.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 --2- 172.21.15.16:0/3583312222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efe54108620 0x7efe5419e980 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:10.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 >> 172.21.15.16:0/3583312222 conn(0x7efe540775e0 msgr2=0x7efe5410cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:10.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 shutdown_connections 2024-04-13T03:31:10.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:10.255+0000 7efe59a00640 1 -- 172.21.15.16:0/3583312222 wait complete. 2024-04-13T03:31:10.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:10 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3583312222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:10.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:10 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3583312222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:10.609 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:10 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3583312222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:10.609 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:31:10 smithi188 systemd[1]: Starting Ceph mds.f for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:31:10.869 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:31:10 smithi188 podman[67099]: 2024-04-13 03:31:10.511818616 +0000 UTC m=+0.017690163 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:31:10.869 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:31:10 smithi188 podman[67099]: 2024-04-13 03:31:10.615606012 +0000 UTC m=+0.121477553 container create 2e1bc20a2d707fe275f8c8ced94e86783c56f89d9ebfecf4a0f73954a918b2c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-13T03:31:10.869 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:31:10 smithi188 podman[67099]: 2024-04-13 03:31:10.868943708 +0000 UTC m=+0.374815261 container init 2e1bc20a2d707fe275f8c8ced94e86783c56f89d9ebfecf4a0f73954a918b2c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409) 2024-04-13T03:31:11.122 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:31:10 smithi188 podman[67099]: 2024-04-13 03:31:10.871540033 +0000 UTC m=+0.377411577 container start 2e1bc20a2d707fe275f8c8ced94e86783c56f89d9ebfecf4a0f73954a918b2c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, CEPH_POINT_RELEASE=) 2024-04-13T03:31:11.123 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:31:10 smithi188 ceph-mds[67118]: -- 172.21.15.188:0/3648467665 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556ff56e3a40 con 0x556ff56b7000 2024-04-13T03:31:11.123 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:31:10 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f[67114]: starting mds.f at 2024-04-13T03:31:11.123 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:31:10 smithi188 bash[67099]: 2e1bc20a2d707fe275f8c8ced94e86783c56f89d9ebfecf4a0f73954a918b2c5 2024-04-13T03:31:11.123 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:31:10 smithi188 systemd[1]: Started Ceph mds.f for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:11.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:11 smithi016 ceph-mon[31772]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:11.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:11.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:11.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:11.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:11.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:11.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:11 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:11.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:11 smithi188 ceph-mon[36398]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:11.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:11.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:11.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:11.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:11.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:11.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:11 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:11.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:11 smithi119 ceph-mon[35835]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:11.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:11.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:11.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:11.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:11.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:11.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:11 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:12.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.509+0000 7f947f200640 1 -- 172.21.15.16:0/2207853353 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94801028f0 msgr2=0x7f9480109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:12.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.509+0000 7f947f200640 1 --2- 172.21.15.16:0/2207853353 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94801028f0 0x7f9480109180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9478009be0 tx=0x7f947802bef0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.509+0000 7f947f200640 1 -- 172.21.15.16:0/2207853353 shutdown_connections 2024-04-13T03:31:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.509+0000 7f947f200640 1 --2- 172.21.15.16:0/2207853353 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94801028f0 0x7f9480109180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.509+0000 7f947f200640 1 --2- 172.21.15.16:0/2207853353 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9480101f30 0x7f94801023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.509+0000 7f947f200640 1 --2- 172.21.15.16:0/2207853353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9480100d40 0x7f9480101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.509+0000 7f947f200640 1 -- 172.21.15.16:0/2207853353 >> 172.21.15.16:0/2207853353 conn(0x7f94800fc2d0 msgr2=0x7f94800fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.509+0000 7f947f200640 1 -- 172.21.15.16:0/2207853353 shutdown_connections 2024-04-13T03:31:12.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.509+0000 7f947f200640 1 -- 172.21.15.16:0/2207853353 wait complete. 2024-04-13T03:31:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.510+0000 7f947f200640 1 Processor -- start 2024-04-13T03:31:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.510+0000 7f947f200640 1 -- start start 2024-04-13T03:31:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9480100d40 0x7f948019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947f200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9480101f30 0x7f948019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94801028f0 0x7f94801a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f948010c9f0 con 0x7f9480100d40 2024-04-13T03:31:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947f200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f948010c870 con 0x7f9480101f30 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f948010cb70 con 0x7f94801028f0 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9480100d40 0x7f948019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9480100d40 0x7f948019a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60864/0 (socket says 172.21.15.16:60864) 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947de00640 1 -- 172.21.15.16:0/4002291431 learned_addr learned my addr 172.21.15.16:0/4002291431 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947d400640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9480101f30 0x7f948019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947e800640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94801028f0 0x7f94801a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947de00640 1 -- 172.21.15.16:0/4002291431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94801028f0 msgr2=0x7f94801a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947de00640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94801028f0 0x7f94801a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947de00640 1 -- 172.21.15.16:0/4002291431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9480101f30 msgr2=0x7f948019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947de00640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9480101f30 0x7f948019a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.511+0000 7f947de00640 1 -- 172.21.15.16:0/4002291431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94801a1f30 con 0x7f9480100d40 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.512+0000 7f947de00640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9480100d40 0x7f948019a310 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f947000e9a0 tx=0x7f947000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.512+0000 7f9476a00640 1 -- 172.21.15.16:0/4002291431 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947000cd80 con 0x7f9480100d40 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.512+0000 7f9476a00640 1 -- 172.21.15.16:0/4002291431 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9470010910 con 0x7f9480100d40 2024-04-13T03:31:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.512+0000 7f9476a00640 1 -- 172.21.15.16:0/4002291431 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470018a30 con 0x7f9480100d40 2024-04-13T03:31:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.512+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94801a2220 con 0x7f9480100d40 2024-04-13T03:31:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.512+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94801a2720 con 0x7f9480100d40 2024-04-13T03:31:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.513+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94801a29e0 con 0x7f9480100d40 2024-04-13T03:31:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.515+0000 7f9476a00640 1 -- 172.21.15.16:0/4002291431 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9470010ac0 con 0x7f9480100d40 2024-04-13T03:31:12.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.517+0000 7f9476a00640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f94640a3c30 0x7f94640a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:12.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.517+0000 7f9476a00640 1 -- 172.21.15.16:0/4002291431 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f94700d0300 con 0x7f9480100d40 2024-04-13T03:31:12.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.517+0000 7f947d400640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f94640a3c30 0x7f94640a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:12.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.518+0000 7f947d400640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f94640a3c30 0x7f94640a60f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f948019b820 tx=0x7f946c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:12.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.521+0000 7f9476a00640 1 -- 172.21.15.16:0/4002291431 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9470053ff0 con 0x7f9480100d40 2024-04-13T03:31:12.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:12 smithi016 ceph-mon[31772]: Deploying daemon mds.c on smithi188 2024-04-13T03:31:12.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:12 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6834/566226134,v1:172.21.15.188:6835/566226134] up:boot 2024-04-13T03:31:12.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:12 smithi016 ceph-mon[31772]: fsmap 6 up:standby 2024-04-13T03:31:12.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:12 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-13T03:31:12.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:12 smithi188 ceph-mon[36398]: Deploying daemon mds.c on smithi188 2024-04-13T03:31:12.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:12 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6834/566226134,v1:172.21.15.188:6835/566226134] up:boot 2024-04-13T03:31:12.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:12 smithi188 ceph-mon[36398]: fsmap 6 up:standby 2024-04-13T03:31:12.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:12 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-13T03:31:12.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.770+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94801a2cd0 con 0x7f9480100d40 2024-04-13T03:31:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.771+0000 7f9476a00640 1 -- 172.21.15.16:0/4002291431 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f947009d0e0 con 0x7f9480100d40 2024-04-13T03:31:12.771 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:12.771 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-13T03:31:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2024-04-13T03:31:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.773+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f94640a3c30 msgr2=0x7f94640a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.773+0000 7f947f200640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f94640a3c30 0x7f94640a60f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f948019b820 tx=0x7f946c005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:12.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.773+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9480100d40 msgr2=0x7f948019a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:12.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.773+0000 7f947f200640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9480100d40 0x7f948019a310 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f947000e9a0 tx=0x7f947000ee60 comp rx=0 tx=0).stop 2024-04-13T03:31:12.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.774+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 shutdown_connections 2024-04-13T03:31:12.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.774+0000 7f947f200640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f94640a3c30 0x7f94640a60f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.774+0000 7f947f200640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94801028f0 0x7f94801a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.774+0000 7f947f200640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9480101f30 0x7f948019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.774+0000 7f947f200640 1 --2- 172.21.15.16:0/4002291431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9480100d40 0x7f948019a310 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.774+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 >> 172.21.15.16:0/4002291431 conn(0x7f94800fc2d0 msgr2=0x7f94800fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:12.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.774+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 shutdown_connections 2024-04-13T03:31:12.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:12.774+0000 7f947f200640 1 -- 172.21.15.16:0/4002291431 wait complete. 2024-04-13T03:31:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:12 smithi119 ceph-mon[35835]: Deploying daemon mds.c on smithi188 2024-04-13T03:31:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:12 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6834/566226134,v1:172.21.15.188:6835/566226134] up:boot 2024-04-13T03:31:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:12 smithi119 ceph-mon[35835]: fsmap 6 up:standby 2024-04-13T03:31:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:12 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-13T03:31:13.360 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:31:13 smithi188 systemd[1]: Starting Ceph mds.c for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:31:13.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:13 smithi188 ceph-mon[36398]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:13.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:13 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4002291431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:13.623 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:31:13 smithi188 podman[67409]: 2024-04-13 03:31:13.405323049 +0000 UTC m=+0.017920070 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:31:13.623 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:31:13 smithi188 podman[67409]: 2024-04-13 03:31:13.522204108 +0000 UTC m=+0.134801110 container create ea3859d0b276d4d7e081600838cee7691d27e1397373de852d717f79d0b8d7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-13T03:31:13.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:13 smithi119 ceph-mon[35835]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:13.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:13 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4002291431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:13.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:13 smithi016 ceph-mon[31772]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:13.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:13 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4002291431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:13.885 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:31:13 smithi188 podman[67409]: 2024-04-13 03:31:13.858388912 +0000 UTC m=+0.470985920 container init ea3859d0b276d4d7e081600838cee7691d27e1397373de852d717f79d0b8d7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-13T03:31:13.885 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:31:13 smithi188 podman[67409]: 2024-04-13 03:31:13.861139787 +0000 UTC m=+0.473736804 container start ea3859d0b276d4d7e081600838cee7691d27e1397373de852d717f79d0b8d7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-13T03:31:13.885 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:31:13 smithi188 ceph-mds[67429]: -- 172.21.15.188:0/1679434720 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e41a793a40 con 0x55e41a766800 2024-04-13T03:31:14.372 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:31:13 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c[67425]: starting mds.c at 2024-04-13T03:31:14.372 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:31:13 smithi188 bash[67409]: ea3859d0b276d4d7e081600838cee7691d27e1397373de852d717f79d0b8d7ee 2024-04-13T03:31:14.373 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:31:13 smithi188 systemd[1]: Started Ceph mds.c for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:14.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:15.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.023+0000 7f339be00640 1 -- 172.21.15.16:0/1025033167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f339c109630 msgr2=0x7f339c109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:15.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.023+0000 7f339be00640 1 --2- 172.21.15.16:0/1025033167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f339c109630 0x7f339c109ab0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3384009860 tx=0x7f338402bb70 comp rx=0 tx=0).stop 2024-04-13T03:31:15.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.023+0000 7f339be00640 1 -- 172.21.15.16:0/1025033167 shutdown_connections 2024-04-13T03:31:15.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.023+0000 7f339be00640 1 --2- 172.21.15.16:0/1025033167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339c109ff0 0x7f339c110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.023+0000 7f339be00640 1 --2- 172.21.15.16:0/1025033167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f339c109630 0x7f339c109ab0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.023+0000 7f339be00640 1 --2- 172.21.15.16:0/1025033167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f339c108440 0x7f339c108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.023+0000 7f339be00640 1 -- 172.21.15.16:0/1025033167 >> 172.21.15.16:0/1025033167 conn(0x7f339c077400 msgr2=0x7f339c079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:15.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.024+0000 7f339be00640 1 -- 172.21.15.16:0/1025033167 shutdown_connections 2024-04-13T03:31:15.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.024+0000 7f339be00640 1 -- 172.21.15.16:0/1025033167 wait complete. 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.024+0000 7f339be00640 1 Processor -- start 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339be00640 1 -- start start 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339be00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f339c108440 0x7f339c19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f339c109630 0x7f339c19ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339c109ff0 0x7f339c1a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339be00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f339c1140d0 con 0x7f339c108440 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339be00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f339c113f50 con 0x7f339c109630 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339be00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f339c114250 con 0x7f339c109ff0 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f339c108440 0x7f339c19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f339c108440 0x7f339c19e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60882/0 (socket says 172.21.15.16:60882) 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339aa00640 1 -- 172.21.15.16:0/3174384790 learned_addr learned my addr 172.21.15.16:0/3174384790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:15.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339a000640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f339c109630 0x7f339c19ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.025+0000 7f339b400640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339c109ff0 0x7f339c1a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f339aa00640 1 -- 172.21.15.16:0/3174384790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339c109ff0 msgr2=0x7f339c1a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f339aa00640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339c109ff0 0x7f339c1a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f339aa00640 1 -- 172.21.15.16:0/3174384790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f339c109630 msgr2=0x7f339c19ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f339aa00640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f339c109630 0x7f339c19ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f339aa00640 1 -- 172.21.15.16:0/3174384790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339c1a6340 con 0x7f339c108440 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f339aa00640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f339c108440 0x7f339c19e720 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f338c00c3f0 tx=0x7f338c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f3393400640 1 -- 172.21.15.16:0/3174384790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c01b070 con 0x7f339c108440 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f3393400640 1 -- 172.21.15.16:0/3174384790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f338c00bb10 con 0x7f339c108440 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f339c1a6630 con 0x7f339c108440 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f3393400640 1 -- 172.21.15.16:0/3174384790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c016d80 con 0x7f339c108440 2024-04-13T03:31:15.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.026+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f339c1a6ba0 con 0x7f339c108440 2024-04-13T03:31:15.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.028+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f339c10f760 con 0x7f339c108440 2024-04-13T03:31:15.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.030+0000 7f3393400640 1 -- 172.21.15.16:0/3174384790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f338c01e070 con 0x7f339c108440 2024-04-13T03:31:15.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.031+0000 7f3393400640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f33800a3c30 0x7f33800a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:15.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.031+0000 7f3393400640 1 -- 172.21.15.16:0/3174384790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f338c0cfe00 con 0x7f339c108440 2024-04-13T03:31:15.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.032+0000 7f339a000640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f33800a3c30 0x7f33800a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:15.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.032+0000 7f339a000640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f33800a3c30 0x7f33800a60f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3384009830 tx=0x7f3384034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:15.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.036+0000 7f3393400640 1 -- 172.21.15.16:0/3174384790 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f338c09c590 con 0x7f339c108440 2024-04-13T03:31:15.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.286+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f339c02d5e0 con 0x7f339c108440 2024-04-13T03:31:15.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.287+0000 7f3393400640 1 -- 172.21.15.16:0/3174384790 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f338c0a0890 con 0x7f339c108440 2024-04-13T03:31:15.288 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:15.288 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-13T03:31:15.288 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-13T03:31:15.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f33800a3c30 msgr2=0x7f33800a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:15.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f33800a3c30 0x7f33800a60f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3384009830 tx=0x7f3384034ab0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f339c108440 msgr2=0x7f339c19e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:15.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f339c108440 0x7f339c19e720 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f338c00c3f0 tx=0x7f338c009f90 comp rx=0 tx=0).stop 2024-04-13T03:31:15.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 shutdown_connections 2024-04-13T03:31:15.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f33800a3c30 0x7f33800a60f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f339c109ff0 0x7f339c1a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f339c109630 0x7f339c19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 --2- 172.21.15.16:0/3174384790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f339c108440 0x7f339c19e720 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:15.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 >> 172.21.15.16:0/3174384790 conn(0x7f339c077400 msgr2=0x7f339c10c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:15.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 shutdown_connections 2024-04-13T03:31:15.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:15.290+0000 7f339be00640 1 -- 172.21.15.16:0/3174384790 wait complete. 2024-04-13T03:31:15.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: Deploying daemon mds.h on smithi119 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] up:boot 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: fsmap 7 up:standby 2024-04-13T03:31:15.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:14 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-13T03:31:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:15.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:15.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: Deploying daemon mds.h on smithi119 2024-04-13T03:31:15.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] up:boot 2024-04-13T03:31:15.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: fsmap 7 up:standby 2024-04-13T03:31:15.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:14 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-13T03:31:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: Deploying daemon mds.h on smithi119 2024-04-13T03:31:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] up:boot 2024-04-13T03:31:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: fsmap 7 up:standby 2024-04-13T03:31:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:14 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-13T03:31:16.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:15 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3174384790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:16.322 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:31:16 smithi119 systemd[1]: Starting Ceph mds.h for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:31:16.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:15 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3174384790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:16.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:15 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3174384790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:16.701 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:31:16 smithi119 podman[65899]: 2024-04-13 03:31:16.289552352 +0000 UTC m=+0.017961341 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:31:16.701 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:31:16 smithi119 podman[65899]: 2024-04-13 03:31:16.432367943 +0000 UTC m=+0.160776965 container create cb36ac31d9928920e7ef19c9b148f8be43256f352f6b260e6ff678c42888674b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:31:16.702 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:31:16 smithi119 podman[65899]: 2024-04-13 03:31:16.701743649 +0000 UTC m=+0.430152628 container init cb36ac31d9928920e7ef19c9b148f8be43256f352f6b260e6ff678c42888674b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True) 2024-04-13T03:31:17.002 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:31:16 smithi119 podman[65899]: 2024-04-13 03:31:16.704291928 +0000 UTC m=+0.432700907 container start cb36ac31d9928920e7ef19c9b148f8be43256f352f6b260e6ff678c42888674b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-13T03:31:17.002 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:31:16 smithi119 ceph-mds[65918]: -- 172.21.15.119:0/1691793291 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558bc8d9da40 con 0x558bc8d70800 2024-04-13T03:31:17.002 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:31:16 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h[65914]: starting mds.h at 2024-04-13T03:31:17.002 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:31:16 smithi119 bash[65899]: cb36ac31d9928920e7ef19c9b148f8be43256f352f6b260e6ff678c42888674b 2024-04-13T03:31:17.002 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:31:16 smithi119 systemd[1]: Started Ceph mds.h for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:17.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:17 smithi119 ceph-mon[35835]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:17 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:17 smithi016 ceph-mon[31772]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:17.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:17 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:17 smithi188 ceph-mon[36398]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:17.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:17 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:17.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.548+0000 7fc7ca400640 1 -- 172.21.15.16:0/3295678060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7c4105230 msgr2=0x7fc7c41056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.548+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3295678060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7c4105230 0x7fc7c41056b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009860 tx=0x7fc7b002bb70 comp rx=0 tx=0).stop 2024-04-13T03:31:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.548+0000 7fc7ca400640 1 -- 172.21.15.16:0/3295678060 shutdown_connections 2024-04-13T03:31:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.548+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3295678060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7c4105bf0 0x7fc7c410c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.548+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3295678060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7c4105230 0x7fc7c41056b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.548+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3295678060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7c4104040 0x7fc7c4104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.548+0000 7fc7ca400640 1 -- 172.21.15.16:0/3295678060 >> 172.21.15.16:0/3295678060 conn(0x7fc7c40ff5d0 msgr2=0x7fc7c41019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.548+0000 7fc7ca400640 1 -- 172.21.15.16:0/3295678060 shutdown_connections 2024-04-13T03:31:17.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.548+0000 7fc7ca400640 1 -- 172.21.15.16:0/3295678060 wait complete. 2024-04-13T03:31:17.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.550+0000 7fc7ca400640 1 Processor -- start 2024-04-13T03:31:17.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.550+0000 7fc7ca400640 1 -- start start 2024-04-13T03:31:17.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.550+0000 7fc7ca400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7c4104040 0x7fc7c419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.550+0000 7fc7ca400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7c4105230 0x7fc7c419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.550+0000 7fc7ca400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7c4105bf0 0x7fc7c41a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.550+0000 7fc7ca400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c410fcd0 con 0x7fc7c4105bf0 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.550+0000 7fc7ca400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c410fb50 con 0x7fc7c4104040 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.550+0000 7fc7ca400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c410fe50 con 0x7fc7c4105230 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7c4105230 0x7fc7c419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c9000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7c4104040 0x7fc7c419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7c4105230 0x7fc7c419a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:56324/0 (socket says 172.21.15.16:56324) 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c3e00640 1 -- 172.21.15.16:0/3925161098 learned_addr learned my addr 172.21.15.16:0/3925161098 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c9a00640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7c4105bf0 0x7fc7c41a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c3e00640 1 -- 172.21.15.16:0/3925161098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7c4104040 msgr2=0x7fc7c419a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c3e00640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7c4104040 0x7fc7c419a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c3e00640 1 -- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7c4105bf0 msgr2=0x7fc7c41a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c3e00640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7c4105bf0 0x7fc7c41a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c3e00640 1 -- 172.21.15.16:0/3925161098 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c41a1f90 con 0x7fc7c4105230 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c9a00640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7c4105bf0 0x7fc7c41a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.551+0000 7fc7c3e00640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7c4105230 0x7fc7c419a8b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00349e0 tx=0x7fc7b0032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.552+0000 7fc7c1600640 1 -- 172.21.15.16:0/3925161098 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b0041030 con 0x7fc7c4105230 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.552+0000 7fc7c1600640 1 -- 172.21.15.16:0/3925161098 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7b002c510 con 0x7fc7c4105230 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.552+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7c41a2220 con 0x7fc7c4105230 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.552+0000 7fc7c1600640 1 -- 172.21.15.16:0/3925161098 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b0039040 con 0x7fc7c4105230 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.552+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7c41a2760 con 0x7fc7c4105230 2024-04-13T03:31:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.553+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7c410b360 con 0x7fc7c4105230 2024-04-13T03:31:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.555+0000 7fc7c1600640 1 -- 172.21.15.16:0/3925161098 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc7b0044070 con 0x7fc7c4105230 2024-04-13T03:31:17.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.557+0000 7fc7c1600640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc7ac0a3c30 0x7fc7ac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:17.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.557+0000 7fc7c1600640 1 -- 172.21.15.16:0/3925161098 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc7b00f1550 con 0x7fc7c4105230 2024-04-13T03:31:17.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.557+0000 7fc7c9000640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc7ac0a3c30 0x7fc7ac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:17.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.558+0000 7fc7c9000640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc7ac0a3c30 0x7fc7ac0a60f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009530 tx=0x7fc7b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:17.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.560+0000 7fc7c1600640 1 -- 172.21.15.16:0/3925161098 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7b00be330 con 0x7fc7c4105230 2024-04-13T03:31:17.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.809+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7c4062bc0 con 0x7fc7c4105230 2024-04-13T03:31:17.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.810+0000 7fc7c1600640 1 -- 172.21.15.16:0/3925161098 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7fc7b00bdc30 con 0x7fc7c4105230 2024-04-13T03:31:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:17.810 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-13T03:31:17.811 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 9 2024-04-13T03:31:17.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc7ac0a3c30 msgr2=0x7fc7ac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc7ac0a3c30 0x7fc7ac0a60f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009530 tx=0x7fc7b8009260 comp rx=0 tx=0).stop 2024-04-13T03:31:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7c4105230 msgr2=0x7fc7c419a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7c4105230 0x7fc7c419a8b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00349e0 tx=0x7fc7b0032450 comp rx=0 tx=0).stop 2024-04-13T03:31:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 shutdown_connections 2024-04-13T03:31:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc7ac0a3c30 0x7fc7ac0a60f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc7c4105bf0 0x7fc7c41a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7c4105230 0x7fc7c419a8b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 --2- 172.21.15.16:0/3925161098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7c4104040 0x7fc7c419a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 >> 172.21.15.16:0/3925161098 conn(0x7fc7c40ff5d0 msgr2=0x7fc7c4108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:17.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 shutdown_connections 2024-04-13T03:31:17.815 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:17.813+0000 7fc7ca400640 1 -- 172.21.15.16:0/3925161098 wait complete. 2024-04-13T03:31:18.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:18 smithi016 ceph-mon[31772]: Deploying daemon mds.i on smithi188 2024-04-13T03:31:18.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:18 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] up:boot 2024-04-13T03:31:18.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:18 smithi016 ceph-mon[31772]: fsmap 8 up:standby 2024-04-13T03:31:18.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:18 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-13T03:31:18.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:18 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3925161098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:18.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:18 smithi188 ceph-mon[36398]: Deploying daemon mds.i on smithi188 2024-04-13T03:31:18.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:18 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] up:boot 2024-04-13T03:31:18.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:18 smithi188 ceph-mon[36398]: fsmap 8 up:standby 2024-04-13T03:31:18.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:18 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-13T03:31:18.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:18 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3925161098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:18.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:18 smithi119 ceph-mon[35835]: Deploying daemon mds.i on smithi188 2024-04-13T03:31:18.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:18 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] up:boot 2024-04-13T03:31:18.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:18 smithi119 ceph-mon[35835]: fsmap 8 up:standby 2024-04-13T03:31:18.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:18 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-13T03:31:18.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:18 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3925161098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:19.319 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:19 smithi188 ceph-mon[36398]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:19.319 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:31:19 smithi188 systemd[1]: Starting Ceph mds.i for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:31:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:19 smithi119 ceph-mon[35835]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:19.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:19 smithi016 ceph-mon[31772]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:19.623 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:31:19 smithi188 podman[67719]: 2024-04-13 03:31:19.221394306 +0000 UTC m=+0.017829077 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:31:19.623 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:31:19 smithi188 podman[67719]: 2024-04-13 03:31:19.327812404 +0000 UTC m=+0.124247174 container create cc5aff15da9c5d83b4a74f501f89ebee995a1c616622e77105e0b641e476da30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:31:19.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.059+0000 7f82ea200640 1 -- 172.21.15.16:0/536165192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82e4105410 msgr2=0x7f82e4105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.059+0000 7f82ea200640 1 --2- 172.21.15.16:0/536165192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82e4105410 0x7f82e4105890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009900 tx=0x7f82d002bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.059+0000 7f82ea200640 1 -- 172.21.15.16:0/536165192 shutdown_connections 2024-04-13T03:31:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.059+0000 7f82ea200640 1 --2- 172.21.15.16:0/536165192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4105dd0 0x7f82e410c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.059+0000 7f82ea200640 1 --2- 172.21.15.16:0/536165192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82e4105410 0x7f82e4105890 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.059+0000 7f82ea200640 1 --2- 172.21.15.16:0/536165192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e4104220 0x7f82e4104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.059+0000 7f82ea200640 1 -- 172.21.15.16:0/536165192 >> 172.21.15.16:0/536165192 conn(0x7f82e40ff7b0 msgr2=0x7f82e4101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.059+0000 7f82ea200640 1 -- 172.21.15.16:0/536165192 shutdown_connections 2024-04-13T03:31:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.059+0000 7f82ea200640 1 -- 172.21.15.16:0/536165192 wait complete. 2024-04-13T03:31:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.060+0000 7f82ea200640 1 Processor -- start 2024-04-13T03:31:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.060+0000 7f82ea200640 1 -- start start 2024-04-13T03:31:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82ea200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4104220 0x7f82e419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82ea200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e4105410 0x7f82e419aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82ea200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82e4105dd0 0x7f82e41a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82ea200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82e410feb0 con 0x7f82e4104220 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82ea200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f82e410fd30 con 0x7f82e4105410 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82ea200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f82e4110030 con 0x7f82e4105dd0 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4104220 0x7f82e419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4104220 0x7f82e419a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51052/0 (socket says 172.21.15.16:51052) 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e3400640 1 -- 172.21.15.16:0/2876046062 learned_addr learned my addr 172.21.15.16:0/2876046062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e2a00640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e4105410 0x7f82e419aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e3e00640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82e4105dd0 0x7f82e41a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e3400640 1 -- 172.21.15.16:0/2876046062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82e4105dd0 msgr2=0x7f82e41a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e3400640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82e4105dd0 0x7f82e41a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e3400640 1 -- 172.21.15.16:0/2876046062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e4105410 msgr2=0x7f82e419aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:20.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e3400640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e4105410 0x7f82e419aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.061+0000 7f82e3400640 1 -- 172.21.15.16:0/2876046062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e41a2150 con 0x7f82e4104220 2024-04-13T03:31:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.062+0000 7f82e3400640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4104220 0x7f82e419a530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f82d400e9c0 tx=0x7f82d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.062+0000 7f82dbe00640 1 -- 172.21.15.16:0/2876046062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d400cd80 con 0x7f82e4104220 2024-04-13T03:31:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.062+0000 7f82dbe00640 1 -- 172.21.15.16:0/2876046062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82d40035f0 con 0x7f82e4104220 2024-04-13T03:31:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.062+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e41a2440 con 0x7f82e4104220 2024-04-13T03:31:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.062+0000 7f82dbe00640 1 -- 172.21.15.16:0/2876046062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4010690 con 0x7f82e4104220 2024-04-13T03:31:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.062+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e41a2960 con 0x7f82e4104220 2024-04-13T03:31:20.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.064+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e410b540 con 0x7f82e4104220 2024-04-13T03:31:20.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.065+0000 7f82dbe00640 1 -- 172.21.15.16:0/2876046062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f82d40037a0 con 0x7f82e4104220 2024-04-13T03:31:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.067+0000 7f82dbe00640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82c80a3c30 0x7f82c80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.067+0000 7f82dbe00640 1 -- 172.21.15.16:0/2876046062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f82d40d0390 con 0x7f82e4104220 2024-04-13T03:31:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.067+0000 7f82e2a00640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82c80a3c30 0x7f82c80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.068+0000 7f82e2a00640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82c80a3c30 0x7f82c80a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f82d002c7b0 tx=0x7f82d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:20.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.072+0000 7f82dbe00640 1 -- 172.21.15.16:0/2876046062 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f82d409d170 con 0x7f82e4104220 2024-04-13T03:31:20.122 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:31:19 smithi188 podman[67719]: 2024-04-13 03:31:19.664442838 +0000 UTC m=+0.460877610 container init cc5aff15da9c5d83b4a74f501f89ebee995a1c616622e77105e0b641e476da30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:31:20.123 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:31:19 smithi188 podman[67719]: 2024-04-13 03:31:19.667280017 +0000 UTC m=+0.463714795 container start cc5aff15da9c5d83b4a74f501f89ebee995a1c616622e77105e0b641e476da30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-13T03:31:20.123 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:31:19 smithi188 ceph-mds[67738]: -- 172.21.15.188:0/387403745 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5608d47afa40 con 0x5608d4783000 2024-04-13T03:31:20.123 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:31:19 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i[67734]: starting mds.i at 2024-04-13T03:31:20.123 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:31:19 smithi188 bash[67719]: cc5aff15da9c5d83b4a74f501f89ebee995a1c616622e77105e0b641e476da30 2024-04-13T03:31:20.123 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:31:19 smithi188 systemd[1]: Started Ceph mds.i for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.341+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82e402d5e0 con 0x7f82e4104220 2024-04-13T03:31:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.343+0000 7f82dbe00640 1 -- 172.21.15.16:0/2876046062 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f82d409ca70 con 0x7f82e4104220 2024-04-13T03:31:20.344 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:20.344 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-13T03:31:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 10 2024-04-13T03:31:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82c80a3c30 msgr2=0x7f82c80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82c80a3c30 0x7f82c80a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f82d002c7b0 tx=0x7f82d0005fd0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4104220 msgr2=0x7f82e419a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4104220 0x7f82e419a530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f82d400e9c0 tx=0x7f82d400ee80 comp rx=0 tx=0).stop 2024-04-13T03:31:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 shutdown_connections 2024-04-13T03:31:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82c80a3c30 0x7f82c80a60f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82e4105dd0 0x7f82e41a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e4105410 0x7f82e419aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 --2- 172.21.15.16:0/2876046062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82e4104220 0x7f82e419a530 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 >> 172.21.15.16:0/2876046062 conn(0x7f82e40ff7b0 msgr2=0x7f82e4108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 shutdown_connections 2024-04-13T03:31:20.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:20.346+0000 7f82ea200640 1 -- 172.21.15.16:0/2876046062 wait complete. 2024-04-13T03:31:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:21.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:21.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:21.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:21.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: Deploying daemon mds.g on smithi016 2024-04-13T03:31:21.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] up:boot 2024-04-13T03:31:21.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: fsmap 9 up:standby 2024-04-13T03:31:21.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-13T03:31:21.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:20 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2876046062' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: Deploying daemon mds.g on smithi016 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] up:boot 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: fsmap 9 up:standby 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-13T03:31:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:20 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2876046062' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: Deploying daemon mds.g on smithi016 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] up:boot 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: fsmap 9 up:standby 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-13T03:31:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:20 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2876046062' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:22.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.600+0000 7fd671600640 1 -- 172.21.15.16:0/1180196739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd66c105dd0 msgr2=0x7fd66c10c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.600+0000 7fd671600640 1 --2- 172.21.15.16:0/1180196739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd66c105dd0 0x7fd66c10c660 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd664009900 tx=0x7fd66402bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.600+0000 7fd671600640 1 -- 172.21.15.16:0/1180196739 shutdown_connections 2024-04-13T03:31:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.600+0000 7fd671600640 1 --2- 172.21.15.16:0/1180196739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd66c105dd0 0x7fd66c10c660 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.600+0000 7fd671600640 1 --2- 172.21.15.16:0/1180196739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd66c105410 0x7fd66c105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.600+0000 7fd671600640 1 --2- 172.21.15.16:0/1180196739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd66c104220 0x7fd66c104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.600+0000 7fd671600640 1 -- 172.21.15.16:0/1180196739 >> 172.21.15.16:0/1180196739 conn(0x7fd66c0ff7b0 msgr2=0x7fd66c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.600+0000 7fd671600640 1 -- 172.21.15.16:0/1180196739 shutdown_connections 2024-04-13T03:31:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.600+0000 7fd671600640 1 -- 172.21.15.16:0/1180196739 wait complete. 2024-04-13T03:31:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.601+0000 7fd671600640 1 Processor -- start 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.601+0000 7fd671600640 1 -- start start 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd671600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd66c104220 0x7fd66c19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd671600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd66c105410 0x7fd66c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd671600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd66c105dd0 0x7fd66c1a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd671600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c10feb0 con 0x7fd66c104220 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd671600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c10fd30 con 0x7fd66c105410 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd671600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd66c110030 con 0x7fd66c105dd0 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd66c104220 0x7fd66c19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd66c104220 0x7fd66c19a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51056/0 (socket says 172.21.15.16:51056) 2024-04-13T03:31:22.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66aa00640 1 -- 172.21.15.16:0/1437889169 learned_addr learned my addr 172.21.15.16:0/1437889169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66b400640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd66c105dd0 0x7fd66c1a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66a000640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd66c105410 0x7fd66c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66aa00640 1 -- 172.21.15.16:0/1437889169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd66c105dd0 msgr2=0x7fd66c1a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66aa00640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd66c105dd0 0x7fd66c1a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66aa00640 1 -- 172.21.15.16:0/1437889169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd66c105410 msgr2=0x7fd66c19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66aa00640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd66c105410 0x7fd66c19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.602+0000 7fd66aa00640 1 -- 172.21.15.16:0/1437889169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66c1a20d0 con 0x7fd66c104220 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.603+0000 7fd66aa00640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd66c104220 0x7fd66c19a4b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00eb70 tx=0x7fd65c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.603+0000 7fd663400640 1 -- 172.21.15.16:0/1437889169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c003b70 con 0x7fd66c104220 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.603+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66c1a23c0 con 0x7fd66c104220 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.603+0000 7fd663400640 1 -- 172.21.15.16:0/1437889169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c00ee70 con 0x7fd66c104220 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.603+0000 7fd663400640 1 -- 172.21.15.16:0/1437889169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c010830 con 0x7fd66c104220 2024-04-13T03:31:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.603+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66c1a2910 con 0x7fd66c104220 2024-04-13T03:31:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.604+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd66c10b540 con 0x7fd66c104220 2024-04-13T03:31:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.607+0000 7fd663400640 1 -- 172.21.15.16:0/1437889169 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd65c004a40 con 0x7fd66c104220 2024-04-13T03:31:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.608+0000 7fd663400640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd6500a3bb0 0x7fd6500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.608+0000 7fd663400640 1 -- 172.21.15.16:0/1437889169 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd65c0d0750 con 0x7fd66c104220 2024-04-13T03:31:22.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.608+0000 7fd66a000640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd6500a3bb0 0x7fd6500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:22.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.609+0000 7fd66a000640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd6500a3bb0 0x7fd6500a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd66c19b9c0 tx=0x7fd654005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.611+0000 7fd663400640 1 -- 172.21.15.16:0/1437889169 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd65c09d5d0 con 0x7fd66c104220 2024-04-13T03:31:22.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.883+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd66c02d5e0 con 0x7fd66c104220 2024-04-13T03:31:22.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.884+0000 7fd663400640 1 -- 172.21.15.16:0/1437889169 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7fd65c09ced0 con 0x7fd66c104220 2024-04-13T03:31:22.884 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:22.884 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-13T03:31:22.885 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 10 2024-04-13T03:31:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd6500a3bb0 msgr2=0x7fd6500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd6500a3bb0 0x7fd6500a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd66c19b9c0 tx=0x7fd654005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd66c104220 msgr2=0x7fd66c19a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd66c104220 0x7fd66c19a4b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00eb70 tx=0x7fd65c00c6a0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 shutdown_connections 2024-04-13T03:31:22.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd6500a3bb0 0x7fd6500a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd66c105dd0 0x7fd66c1a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd66c105410 0x7fd66c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 --2- 172.21.15.16:0/1437889169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd66c104220 0x7fd66c19a4b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 >> 172.21.15.16:0/1437889169 conn(0x7fd66c0ff7b0 msgr2=0x7fd66c108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 shutdown_connections 2024-04-13T03:31:22.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:22.887+0000 7fd671600640 1 -- 172.21.15.16:0/1437889169 wait complete. 2024-04-13T03:31:23.248 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:22 smithi016 ceph-mon[31772]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:23.248 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:22 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1437889169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:23.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:22 smithi188 ceph-mon[36398]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:23.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:22 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1437889169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:23.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:22 smithi119 ceph-mon[35835]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:23.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:22 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1437889169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:23.937 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:31:23 smithi016 podman[70167]: 2024-04-13 03:31:23.63970136 +0000 UTC m=+0.156134288 container create 8fc36c5f69e5073d0fa3e8b770c2a6d97b37655c3f9f0cdd71d66a7a44814fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-13T03:31:23.937 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:31:23 smithi016 podman[70167]: 2024-04-13 03:31:23.935076803 +0000 UTC m=+0.451509721 container init 8fc36c5f69e5073d0fa3e8b770c2a6d97b37655c3f9f0cdd71d66a7a44814fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-13T03:31:24.355 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:31:23 smithi016 podman[70167]: 2024-04-13 03:31:23.945051441 +0000 UTC m=+0.461484360 container start 8fc36c5f69e5073d0fa3e8b770c2a6d97b37655c3f9f0cdd71d66a7a44814fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-13T03:31:24.355 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:31:23 smithi016 ceph-mds[70186]: -- 172.21.15.16:0/3135990548 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558a86d1da40 con 0x558a86cf0800 2024-04-13T03:31:24.355 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:31:23 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g[70182]: starting mds.g at 2024-04-13T03:31:24.355 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:31:24 smithi016 bash[70167]: 8fc36c5f69e5073d0fa3e8b770c2a6d97b37655c3f9f0cdd71d66a7a44814fd0 2024-04-13T03:31:24.356 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:31:24 smithi016 systemd[1]: Started Ceph mds.g for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:24.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.110+0000 7fb4b2e00640 1 -- 172.21.15.16:0/3170412888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105bf0 msgr2=0x7fb4ac10c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.110+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/3170412888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105bf0 0x7fb4ac10c480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009ca0 tx=0x7fb4a402c0f0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.111+0000 7fb4b2e00640 1 -- 172.21.15.16:0/3170412888 shutdown_connections 2024-04-13T03:31:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.111+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/3170412888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105bf0 0x7fb4ac10c480 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.111+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/3170412888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4ac105230 0x7fb4ac1056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.111+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/3170412888 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac104040 0x7fb4ac104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.111+0000 7fb4b2e00640 1 -- 172.21.15.16:0/3170412888 >> 172.21.15.16:0/3170412888 conn(0x7fb4ac0ff5d0 msgr2=0x7fb4ac1019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.111+0000 7fb4b2e00640 1 -- 172.21.15.16:0/3170412888 shutdown_connections 2024-04-13T03:31:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.111+0000 7fb4b2e00640 1 -- 172.21.15.16:0/3170412888 wait complete. 2024-04-13T03:31:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.112+0000 7fb4b2e00640 1 Processor -- start 2024-04-13T03:31:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.112+0000 7fb4b2e00640 1 -- start start 2024-04-13T03:31:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.112+0000 7fb4b2e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4ac104040 0x7fb4ac19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.112+0000 7fb4b2e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105230 0x7fb4ac19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:25.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.112+0000 7fb4b2e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac105bf0 0x7fb4ac1a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.112+0000 7fb4b2e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac10fcd0 con 0x7fb4ac105230 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.112+0000 7fb4b2e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac10fb50 con 0x7fb4ac104040 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.112+0000 7fb4b2e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb4ac10fe50 con 0x7fb4ac105bf0 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105230 0x7fb4ac19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105230 0x7fb4ac19a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51110/0 (socket says 172.21.15.16:51110) 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1000640 1 -- 172.21.15.16:0/895215606 learned_addr learned my addr 172.21.15.16:0/895215606 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b2400640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac105bf0 0x7fb4ac1a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1a00640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4ac104040 0x7fb4ac19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1000640 1 -- 172.21.15.16:0/895215606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac105bf0 msgr2=0x7fb4ac1a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1000640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac105bf0 0x7fb4ac1a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1000640 1 -- 172.21.15.16:0/895215606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4ac104040 msgr2=0x7fb4ac19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1000640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4ac104040 0x7fb4ac19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1000640 1 -- 172.21.15.16:0/895215606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4ac1a1ec0 con 0x7fb4ac105230 2024-04-13T03:31:25.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.113+0000 7fb4b1000640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105230 0x7fb4ac19a7e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb49800ea60 tx=0x7fb49800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:25.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.114+0000 7fb4aa000640 1 -- 172.21.15.16:0/895215606 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49800cde0 con 0x7fb4ac105230 2024-04-13T03:31:25.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.114+0000 7fb4aa000640 1 -- 172.21.15.16:0/895215606 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4980035f0 con 0x7fb4ac105230 2024-04-13T03:31:25.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.114+0000 7fb4aa000640 1 -- 172.21.15.16:0/895215606 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498010670 con 0x7fb4ac105230 2024-04-13T03:31:25.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.114+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4ac1a21b0 con 0x7fb4ac105230 2024-04-13T03:31:25.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.114+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4ac1a26b0 con 0x7fb4ac105230 2024-04-13T03:31:25.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.115+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4ac10b300 con 0x7fb4ac105230 2024-04-13T03:31:25.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.118+0000 7fb4aa000640 1 -- 172.21.15.16:0/895215606 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb4980037a0 con 0x7fb4ac105230 2024-04-13T03:31:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.121+0000 7fb4aa000640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4940a3c30 0x7fb4940a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.121+0000 7fb4aa000640 1 -- 172.21.15.16:0/895215606 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb4980d02b0 con 0x7fb4ac105230 2024-04-13T03:31:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.121+0000 7fb4b1a00640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4940a3c30 0x7fb4940a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:25.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.122+0000 7fb4b1a00640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4940a3c30 0x7fb4940a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005fd0 tx=0x7fb4a000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:25.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.123+0000 7fb4aa000640 1 -- 172.21.15.16:0/895215606 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb49809d120 con 0x7fb4ac105230 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: Deploying daemon mds.a on smithi016 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.16:6838/3959195994,v1:172.21.15.16:6839/3959195994] up:boot 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: fsmap 10 up:standby 2024-04-13T03:31:25.264 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-13T03:31:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.376+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4ac02d5e0 con 0x7fb4ac105230 2024-04-13T03:31:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.378+0000 7fb4aa000640 1 -- 172.21.15.16:0/895215606 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fb49809ca20 con 0x7fb4ac105230 2024-04-13T03:31:25.379 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:25.379 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-13T03:31:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 11 2024-04-13T03:31:25.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4940a3c30 msgr2=0x7fb4940a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:25.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4940a3c30 0x7fb4940a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0005fd0 tx=0x7fb4a000b410 comp rx=0 tx=0).stop 2024-04-13T03:31:25.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105230 msgr2=0x7fb4ac19a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:25.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105230 0x7fb4ac19a7e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb49800ea60 tx=0x7fb49800ef20 comp rx=0 tx=0).stop 2024-04-13T03:31:25.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 shutdown_connections 2024-04-13T03:31:25.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4940a3c30 0x7fb4940a60f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4ac105bf0 0x7fb4ac1a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4ac105230 0x7fb4ac19a7e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 --2- 172.21.15.16:0/895215606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4ac104040 0x7fb4ac19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:25.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 >> 172.21.15.16:0/895215606 conn(0x7fb4ac0ff5d0 msgr2=0x7fb4ac1084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:25.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 shutdown_connections 2024-04-13T03:31:25.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:25.381+0000 7fb4b2e00640 1 -- 172.21.15.16:0/895215606 wait complete. 2024-04-13T03:31:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:25.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:25.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:25.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: Deploying daemon mds.a on smithi016 2024-04-13T03:31:25.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.16:6838/3959195994,v1:172.21.15.16:6839/3959195994] up:boot 2024-04-13T03:31:25.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: fsmap 10 up:standby 2024-04-13T03:31:25.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-13T03:31:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: Deploying daemon mds.a on smithi016 2024-04-13T03:31:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.16:6838/3959195994,v1:172.21.15.16:6839/3959195994] up:boot 2024-04-13T03:31:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: fsmap 10 up:standby 2024-04-13T03:31:25.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-13T03:31:26.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:26 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/895215606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:26.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:26 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/895215606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:26.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:26 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/895215606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:27.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:27.483 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:27 smithi016 ceph-mon[31772]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:27 smithi119 ceph-mon[35835]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:27.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.617+0000 7fa5d5a00640 1 -- 172.21.15.16:0/3053743790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 msgr2=0x7fa5d0102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:27.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.617+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/3053743790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 0x7fa5d0102430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0009860 tx=0x7fa5c002bb70 comp rx=0 tx=0).stop 2024-04-13T03:31:27.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.617+0000 7fa5d5a00640 1 -- 172.21.15.16:0/3053743790 shutdown_connections 2024-04-13T03:31:27.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.617+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/3053743790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0103bc0 0x7fa5d010a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:27.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.617+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/3053743790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d0103200 0x7fa5d0103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:27.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.617+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/3053743790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 0x7fa5d0102430 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:27.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.617+0000 7fa5d5a00640 1 -- 172.21.15.16:0/3053743790 >> 172.21.15.16:0/3053743790 conn(0x7fa5d00fd5a0 msgr2=0x7fa5d00ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:27.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.617+0000 7fa5d5a00640 1 -- 172.21.15.16:0/3053743790 shutdown_connections 2024-04-13T03:31:27.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.617+0000 7fa5d5a00640 1 -- 172.21.15.16:0/3053743790 wait complete. 2024-04-13T03:31:27.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.619+0000 7fa5d5a00640 1 Processor -- start 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.619+0000 7fa5d5a00640 1 -- start start 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.619+0000 7fa5d5a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 0x7fa5d019a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.619+0000 7fa5d5a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0103200 0x7fa5d019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.619+0000 7fa5d5a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d0103bc0 0x7fa5d01a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.619+0000 7fa5d5a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d010dca0 con 0x7fa5d0103200 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.619+0000 7fa5d5a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d010db20 con 0x7fa5d0102010 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.619+0000 7fa5d5a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d010de20 con 0x7fa5d0103bc0 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.619+0000 7fa5cea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 0x7fa5d019a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5cea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 0x7fa5d019a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:44530/0 (socket says 172.21.15.16:44530) 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5cea00640 1 -- 172.21.15.16:0/546205459 learned_addr learned my addr 172.21.15.16:0/546205459 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5cf400640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d0103bc0 0x7fa5d01a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:27.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5ce000640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0103200 0x7fa5d019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:27.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5cea00640 1 -- 172.21.15.16:0/546205459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d0103bc0 msgr2=0x7fa5d01a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5cea00640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d0103bc0 0x7fa5d01a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5cea00640 1 -- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0103200 msgr2=0x7fa5d019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5cea00640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0103200 0x7fa5d019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5cea00640 1 -- 172.21.15.16:0/546205459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d01a2180 con 0x7fa5d0102010 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.620+0000 7fa5cf400640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d0103bc0 0x7fa5d01a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.621+0000 7fa5cea00640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 0x7fa5d019a560 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00349e0 tx=0x7fa5c0032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.621+0000 7fa5c7400640 1 -- 172.21.15.16:0/546205459 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0041020 con 0x7fa5d0102010 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.621+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d01a2410 con 0x7fa5d0102010 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.621+0000 7fa5c7400640 1 -- 172.21.15.16:0/546205459 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c002c9f0 con 0x7fa5d0102010 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.621+0000 7fa5c7400640 1 -- 172.21.15.16:0/546205459 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0039040 con 0x7fa5d0102010 2024-04-13T03:31:27.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.621+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d01a2950 con 0x7fa5d0102010 2024-04-13T03:31:27.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:27 smithi188 ceph-mon[36398]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:27.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.622+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d01a2c70 con 0x7fa5d0102010 2024-04-13T03:31:27.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.625+0000 7fa5c7400640 1 -- 172.21.15.16:0/546205459 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa5c0044020 con 0x7fa5d0102010 2024-04-13T03:31:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.626+0000 7fa5c7400640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa5b40a3bb0 0x7fa5b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.627+0000 7fa5c7400640 1 -- 172.21.15.16:0/546205459 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa5c00f1810 con 0x7fa5d0102010 2024-04-13T03:31:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.627+0000 7fa5ce000640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa5b40a3bb0 0x7fa5b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.628+0000 7fa5ce000640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa5b40a3bb0 0x7fa5b40a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa5d019ba70 tx=0x7fa5bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.631+0000 7fa5c7400640 1 -- 172.21.15.16:0/546205459 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5c004b020 con 0x7fa5d0102010 2024-04-13T03:31:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.909+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5d01a2e40 con 0x7fa5d0102010 2024-04-13T03:31:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.910+0000 7fa5c7400640 1 -- 172.21.15.16:0/546205459 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fa5c00be5f0 con 0x7fa5d0102010 2024-04-13T03:31:27.911 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:27.911 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-13T03:31:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 11 2024-04-13T03:31:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa5b40a3bb0 msgr2=0x7fa5b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa5b40a3bb0 0x7fa5b40a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa5d019ba70 tx=0x7fa5bc005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:27.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 msgr2=0x7fa5d019a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 0x7fa5d019a560 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa5c00349e0 tx=0x7fa5c0032430 comp rx=0 tx=0).stop 2024-04-13T03:31:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 shutdown_connections 2024-04-13T03:31:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa5b40a3bb0 0x7fa5b40a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d0103bc0 0x7fa5d01a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0103200 0x7fa5d019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 --2- 172.21.15.16:0/546205459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5d0102010 0x7fa5d019a560 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 >> 172.21.15.16:0/546205459 conn(0x7fa5d00fd5a0 msgr2=0x7fa5d00741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 shutdown_connections 2024-04-13T03:31:27.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:27.913+0000 7fa5d5a00640 1 -- 172.21.15.16:0/546205459 wait complete. 2024-04-13T03:31:28.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:28 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/546205459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:28.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:28 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/546205459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:28.605 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 13 03:31:28 smithi016 systemd[1]: Started Ceph mds.a for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:28.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:28 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/546205459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:29.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:29.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:29.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:29.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:29.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:29.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:29.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:29.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: Deploying daemon mds.b on smithi119 2024-04-13T03:31:29.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.16:6840/701411811,v1:172.21.15.16:6841/701411811] up:boot 2024-04-13T03:31:29.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: fsmap 11 up:standby 2024-04-13T03:31:29.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: Deploying daemon mds.b on smithi119 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.16:6840/701411811,v1:172.21.15.16:6841/701411811] up:boot 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: fsmap 11 up:standby 2024-04-13T03:31:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-13T03:31:29.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: Deploying daemon mds.b on smithi119 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.16:6840/701411811,v1:172.21.15.16:6841/701411811] up:boot 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: fsmap 11 up:standby 2024-04-13T03:31:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-13T03:31:29.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:30.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.166+0000 7fe416600640 1 -- 172.21.15.16:0/2498351924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109ff0 msgr2=0x7fe410110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.166+0000 7fe416600640 1 --2- 172.21.15.16:0/2498351924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109ff0 0x7fe410110880 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe408009ca0 tx=0x7fe40802c100 comp rx=0 tx=0).stop 2024-04-13T03:31:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.166+0000 7fe416600640 1 -- 172.21.15.16:0/2498351924 shutdown_connections 2024-04-13T03:31:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.166+0000 7fe416600640 1 --2- 172.21.15.16:0/2498351924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109ff0 0x7fe410110880 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.166+0000 7fe416600640 1 --2- 172.21.15.16:0/2498351924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe410109630 0x7fe410109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.166+0000 7fe416600640 1 --2- 172.21.15.16:0/2498351924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe410108440 0x7fe410108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.166+0000 7fe416600640 1 -- 172.21.15.16:0/2498351924 >> 172.21.15.16:0/2498351924 conn(0x7fe410077400 msgr2=0x7fe410079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.166+0000 7fe416600640 1 -- 172.21.15.16:0/2498351924 shutdown_connections 2024-04-13T03:31:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.166+0000 7fe416600640 1 -- 172.21.15.16:0/2498351924 wait complete. 2024-04-13T03:31:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.167+0000 7fe416600640 1 Processor -- start 2024-04-13T03:31:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe416600640 1 -- start start 2024-04-13T03:31:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe416600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe410108440 0x7fe41019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe416600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109630 0x7fe41019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe416600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe410109ff0 0x7fe4101a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe416600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4101140d0 con 0x7fe410109630 2024-04-13T03:31:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe416600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe410113f50 con 0x7fe410108440 2024-04-13T03:31:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe416600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe410114250 con 0x7fe410109ff0 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe40fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109630 0x7fe41019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe415200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe410108440 0x7fe41019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe40fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109630 0x7fe41019ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42230/0 (socket says 172.21.15.16:42230) 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe415200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe410108440 0x7fe41019e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:44560/0 (socket says 172.21.15.16:44560) 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe40fe00640 1 -- 172.21.15.16:0/2006926442 learned_addr learned my addr 172.21.15.16:0/2006926442 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.168+0000 7fe415c00640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe410109ff0 0x7fe4101a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe415200640 1 -- 172.21.15.16:0/2006926442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe410109ff0 msgr2=0x7fe4101a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe415200640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe410109ff0 0x7fe4101a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe415200640 1 -- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109630 msgr2=0x7fe41019ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe415200640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109630 0x7fe41019ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe415200640 1 -- 172.21.15.16:0/2006926442 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4101a6370 con 0x7fe410108440 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe40fe00640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109630 0x7fe41019ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe415c00640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe410109ff0 0x7fe4101a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe415200640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe410108440 0x7fe41019e750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe40400e9c0 tx=0x7fe40400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe40d600640 1 -- 172.21.15.16:0/2006926442 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40400ccc0 con 0x7fe410108440 2024-04-13T03:31:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe40d600640 1 -- 172.21.15.16:0/2006926442 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4040035f0 con 0x7fe410108440 2024-04-13T03:31:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.169+0000 7fe40d600640 1 -- 172.21.15.16:0/2006926442 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40400ce60 con 0x7fe410108440 2024-04-13T03:31:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.170+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4101a6660 con 0x7fe410108440 2024-04-13T03:31:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.170+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4101a6b60 con 0x7fe410108440 2024-04-13T03:31:30.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.171+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe41010f760 con 0x7fe410108440 2024-04-13T03:31:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.173+0000 7fe40d600640 1 -- 172.21.15.16:0/2006926442 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe4040037a0 con 0x7fe410108440 2024-04-13T03:31:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.175+0000 7fe40d600640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3f80a3c30 0x7fe3f80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.175+0000 7fe40d600640 1 -- 172.21.15.16:0/2006926442 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe4040d02c0 con 0x7fe410108440 2024-04-13T03:31:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.175+0000 7fe40fe00640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3f80a3c30 0x7fe3f80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.176+0000 7fe40fe00640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3f80a3c30 0x7fe3f80a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe41019fc60 tx=0x7fe3fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:30.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.179+0000 7fe40d600640 1 -- 172.21.15.16:0/2006926442 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe40409d0a0 con 0x7fe410108440 2024-04-13T03:31:30.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.433+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe41002d5e0 con 0x7fe410108440 2024-04-13T03:31:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.434+0000 7fe40d600640 1 -- 172.21.15.16:0/2006926442 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7fe40409c9a0 con 0x7fe410108440 2024-04-13T03:31:30.435 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:30.435 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-13T03:31:30.435 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 12 2024-04-13T03:31:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3f80a3c30 msgr2=0x7fe3f80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3f80a3c30 0x7fe3f80a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe41019fc60 tx=0x7fe3fc005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe410108440 msgr2=0x7fe41019e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe410108440 0x7fe41019e750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe40400e9c0 tx=0x7fe40400ee80 comp rx=0 tx=0).stop 2024-04-13T03:31:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 shutdown_connections 2024-04-13T03:31:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3f80a3c30 0x7fe3f80a60f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe410109ff0 0x7fe4101a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe410109630 0x7fe41019ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 --2- 172.21.15.16:0/2006926442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe410108440 0x7fe41019e750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.437+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 >> 172.21.15.16:0/2006926442 conn(0x7fe410077400 msgr2=0x7fe41010c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.438+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 shutdown_connections 2024-04-13T03:31:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:30.438+0000 7fe416600640 1 -- 172.21.15.16:0/2006926442 wait complete. 2024-04-13T03:31:30.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:30 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2006926442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:30.821 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:30 smithi119 systemd[1]: Starting Ceph mds.b for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:31:30.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:30 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2006926442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:30.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:30 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2006926442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:31.148 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:30 smithi119 podman[66207]: 2024-04-13 03:31:30.895855687 +0000 UTC m=+0.111385531 container create d80ab5e311a12f00a3b9576118b3ebc911cb90b5bbc8870aa02370f450d27f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:31:31.149 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:30 smithi119 podman[66207]: 2024-04-13 03:31:30.802430501 +0000 UTC m=+0.017960349 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:31:31.496 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:31 smithi119 podman[66207]: 2024-04-13 03:31:31.148628447 +0000 UTC m=+0.364158291 container init d80ab5e311a12f00a3b9576118b3ebc911cb90b5bbc8870aa02370f450d27f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-13T03:31:31.496 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:31 smithi119 podman[66207]: 2024-04-13 03:31:31.151610773 +0000 UTC m=+0.367140620 container start d80ab5e311a12f00a3b9576118b3ebc911cb90b5bbc8870aa02370f450d27f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS) 2024-04-13T03:31:31.496 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:31 smithi119 ceph-mds[66226]: -- 172.21.15.119:0/920040251 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555b46b2da40 con 0x555b46b00c00 2024-04-13T03:31:31.496 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:31 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: starting mds.b at 2024-04-13T03:31:31.496 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:31 smithi119 bash[66207]: d80ab5e311a12f00a3b9576118b3ebc911cb90b5bbc8870aa02370f450d27f47 2024-04-13T03:31:31.496 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:31 smithi119 systemd[1]: Started Ceph mds.b for e6bf0314-f944-11ee-bc8d-c7b262605968. 2024-04-13T03:31:31.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:31 smithi119 ceph-mon[35835]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:31.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:31 smithi016 ceph-mon[31772]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:31.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:31 smithi188 ceph-mon[36398]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:31.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:31.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:32.459 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:32.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.680+0000 7f539f200640 1 -- 172.21.15.16:0/4236504064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0100d40 msgr2=0x7f53a0101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:32.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.680+0000 7f539f200640 1 --2- 172.21.15.16:0/4236504064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0100d40 0x7f53a0101160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5390009860 tx=0x7f539002bb70 comp rx=0 tx=0).stop 2024-04-13T03:31:32.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.680+0000 7f539f200640 1 -- 172.21.15.16:0/4236504064 shutdown_connections 2024-04-13T03:31:32.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.680+0000 7f539f200640 1 --2- 172.21.15.16:0/4236504064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53a01028f0 0x7f53a0109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.680+0000 7f539f200640 1 --2- 172.21.15.16:0/4236504064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0101f30 0x7f53a01023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.680+0000 7f539f200640 1 --2- 172.21.15.16:0/4236504064 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0100d40 0x7f53a0101160 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.680+0000 7f539f200640 1 -- 172.21.15.16:0/4236504064 >> 172.21.15.16:0/4236504064 conn(0x7f53a00fc290 msgr2=0x7f53a00fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.680+0000 7f539f200640 1 -- 172.21.15.16:0/4236504064 shutdown_connections 2024-04-13T03:31:32.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.680+0000 7f539f200640 1 -- 172.21.15.16:0/4236504064 wait complete. 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.681+0000 7f539f200640 1 Processor -- start 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.681+0000 7f539f200640 1 -- start start 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539f200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0100d40 0x7f53a019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0101f30 0x7f53a019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53a01028f0 0x7f53a01a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f53a010c9c0 con 0x7f53a0101f30 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539f200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f53a010c840 con 0x7f53a0100d40 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f53a010cb40 con 0x7f53a01028f0 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0101f30 0x7f53a019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0100d40 0x7f53a019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:32.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539d400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0101f30 0x7f53a019a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42250/0 (socket says 172.21.15.16:42250) 2024-04-13T03:31:32.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539d400640 1 -- 172.21.15.16:0/2131316759 learned_addr learned my addr 172.21.15.16:0/2131316759 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:32.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539e800640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53a01028f0 0x7f53a01a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:32.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539de00640 1 -- 172.21.15.16:0/2131316759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53a01028f0 msgr2=0x7f53a01a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:32.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.682+0000 7f539de00640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53a01028f0 0x7f53a01a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:32.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.683+0000 7f539de00640 1 -- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0101f30 msgr2=0x7f53a019a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:32.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.683+0000 7f539de00640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0101f30 0x7f53a019a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:32.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.683+0000 7f539de00640 1 -- 172.21.15.16:0/2131316759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a01a2000 con 0x7f53a0100d40 2024-04-13T03:31:32.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.683+0000 7f539d400640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0101f30 0x7f53a019a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:32.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.683+0000 7f539de00640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0100d40 0x7f53a019a480 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f539002c0b0 tx=0x7f5390034760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:32.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.683+0000 7f5396a00640 1 -- 172.21.15.16:0/2131316759 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390039040 con 0x7f53a0100d40 2024-04-13T03:31:32.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.683+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a01a2290 con 0x7f53a0100d40 2024-04-13T03:31:32.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.683+0000 7f5396a00640 1 -- 172.21.15.16:0/2131316759 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f539002c510 con 0x7f53a0100d40 2024-04-13T03:31:32.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.683+0000 7f5396a00640 1 -- 172.21.15.16:0/2131316759 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390032bf0 con 0x7f53a0100d40 2024-04-13T03:31:32.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.684+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a01a27a0 con 0x7f53a0100d40 2024-04-13T03:31:32.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.685+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a01a2ac0 con 0x7f53a0100d40 2024-04-13T03:31:32.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.688+0000 7f5396a00640 1 -- 172.21.15.16:0/2131316759 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5390044020 con 0x7f53a0100d40 2024-04-13T03:31:32.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.690+0000 7f5396a00640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f53840a3bb0 0x7f53840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.690+0000 7f5396a00640 1 -- 172.21.15.16:0/2131316759 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f53900f0ff0 con 0x7f53a0100d40 2024-04-13T03:31:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.690+0000 7f539d400640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f53840a3bb0 0x7f53840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:32.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.691+0000 7f5396a00640 1 -- 172.21.15.16:0/2131316759 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f539004b020 con 0x7f53a0100d40 2024-04-13T03:31:32.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.691+0000 7f539d400640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f53840a3bb0 0x7f53840a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f538c0095f0 tx=0x7f538c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:32.747 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:32 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] up:boot 2024-04-13T03:31:32.747 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:32 smithi119 ceph-mon[35835]: fsmap 12 up:standby 2024-04-13T03:31:32.747 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:32 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-13T03:31:32.747 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:32 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:31:32.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:32 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] up:boot 2024-04-13T03:31:32.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:32 smithi016 ceph-mon[31772]: fsmap 12 up:standby 2024-04-13T03:31:32.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:32 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-13T03:31:32.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:32 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:31:32.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:32 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] up:boot 2024-04-13T03:31:32.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:32 smithi188 ceph-mon[36398]: fsmap 12 up:standby 2024-04-13T03:31:32.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:32 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-13T03:31:32.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:32 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:31:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.945+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53a01a2c90 con 0x7f53a0100d40 2024-04-13T03:31:32.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.947+0000 7f5396a00640 1 -- 172.21.15.16:0/2131316759 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7f53900d5900 con 0x7f53a0100d40 2024-04-13T03:31:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:32.949 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-13T03:31:32.949 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f53840a3bb0 msgr2=0x7f53840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f53840a3bb0 0x7f53840a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f538c0095f0 tx=0x7f538c005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0100d40 msgr2=0x7f53a019a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0100d40 0x7f53a019a480 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f539002c0b0 tx=0x7f5390034760 comp rx=0 tx=0).stop 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 shutdown_connections 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f53840a3bb0 0x7f53840a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53a01028f0 0x7f53a01a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53a0101f30 0x7f53a019a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 --2- 172.21.15.16:0/2131316759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0100d40 0x7f53a019a480 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 >> 172.21.15.16:0/2131316759 conn(0x7f53a00fc290 msgr2=0x7f53a00fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 shutdown_connections 2024-04-13T03:31:32.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:32.950+0000 7f539f200640 1 -- 172.21.15.16:0/2131316759 wait complete. 2024-04-13T03:31:32.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:31:33.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.181+0000 7efdd0800640 1 -- 172.21.15.16:0/3403463455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 msgr2=0x7efdc8105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.181+0000 7efdd0800640 1 --2- 172.21.15.16:0/3403463455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 0x7efdc8105890 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009900 tx=0x7efdb802bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:33.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.181+0000 7efdd0800640 1 -- 172.21.15.16:0/3403463455 shutdown_connections 2024-04-13T03:31:33.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.181+0000 7efdd0800640 1 --2- 172.21.15.16:0/3403463455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdc8105dd0 0x7efdc810c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.181+0000 7efdd0800640 1 --2- 172.21.15.16:0/3403463455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 0x7efdc8105890 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.181+0000 7efdd0800640 1 --2- 172.21.15.16:0/3403463455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdc8104220 0x7efdc8104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.181+0000 7efdd0800640 1 -- 172.21.15.16:0/3403463455 >> 172.21.15.16:0/3403463455 conn(0x7efdc80ff7b0 msgr2=0x7efdc8101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:33.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.181+0000 7efdd0800640 1 -- 172.21.15.16:0/3403463455 shutdown_connections 2024-04-13T03:31:33.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.181+0000 7efdd0800640 1 -- 172.21.15.16:0/3403463455 wait complete. 2024-04-13T03:31:33.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.182+0000 7efdd0800640 1 Processor -- start 2024-04-13T03:31:33.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.182+0000 7efdd0800640 1 -- start start 2024-04-13T03:31:33.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdd0800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdc8104220 0x7efdc819a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:33.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdd0800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 0x7efdc819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:33.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdd0800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdc8105dd0 0x7efdc81a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:33.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdd0800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efdc810ff00 con 0x7efdc8104220 2024-04-13T03:31:33.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdd0800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efdc810fd80 con 0x7efdc8105dd0 2024-04-13T03:31:33.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdd0800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efdc8110080 con 0x7efdc8105410 2024-04-13T03:31:33.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdcd400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 0x7efdc819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:33.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdcde00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdc8104220 0x7efdc819a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:33.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdcd400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 0x7efdc819aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:39366/0 (socket says 172.21.15.16:39366) 2024-04-13T03:31:33.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdcd400640 1 -- 172.21.15.16:0/440360245 learned_addr learned my addr 172.21.15.16:0/440360245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:33.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.183+0000 7efdce800640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdc8105dd0 0x7efdc81a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdcd400640 1 -- 172.21.15.16:0/440360245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdc8105dd0 msgr2=0x7efdc81a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdcd400640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdc8105dd0 0x7efdc81a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdcd400640 1 -- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdc8104220 msgr2=0x7efdc819a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdcd400640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdc8104220 0x7efdc819a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdcd400640 1 -- 172.21.15.16:0/440360245 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc81a2140 con 0x7efdc8105410 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdce800640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdc8105dd0 0x7efdc81a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdcde00640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdc8104220 0x7efdc819a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdcd400640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 0x7efdc819aa60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efdb8000c00 tx=0x7efdb802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdc2a00640 1 -- 172.21.15.16:0/440360245 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8043020 con 0x7efdc8105410 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc81a23d0 con 0x7efdc8105410 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdc2a00640 1 -- 172.21.15.16:0/440360245 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdb80054e0 con 0x7efdc8105410 2024-04-13T03:31:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.184+0000 7efdc2a00640 1 -- 172.21.15.16:0/440360245 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8003c40 con 0x7efdc8105410 2024-04-13T03:31:33.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.185+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc81a2970 con 0x7efdc8105410 2024-04-13T03:31:33.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.186+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdc810b4e0 con 0x7efdc8105410 2024-04-13T03:31:33.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.187+0000 7efdc2a00640 1 -- 172.21.15.16:0/440360245 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efdb8046020 con 0x7efdc8105410 2024-04-13T03:31:33.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.189+0000 7efdc2a00640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efdb00a3bb0 0x7efdb00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:33.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.189+0000 7efdc2a00640 1 -- 172.21.15.16:0/440360245 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7efdb80f1fd0 con 0x7efdc8105410 2024-04-13T03:31:33.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.189+0000 7efdcde00640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efdb00a3bb0 0x7efdb00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:33.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.190+0000 7efdcde00640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efdb00a3bb0 0x7efdb00a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efdc8105270 tx=0x7efdbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:33.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.194+0000 7efdc2a00640 1 -- 172.21.15.16:0/440360245 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efdb804d020 con 0x7efdc8105410 2024-04-13T03:31:33.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.412+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efdc8062bc0 con 0x7efdc8105410 2024-04-13T03:31:33.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.414+0000 7efdc2a00640 1 -- 172.21.15.16:0/440360245 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18488 (secure 0 0 0) 0x7efdb80bedb0 con 0x7efdc8105410 2024-04-13T03:31:33.415 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:33.415 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:30:39.665995+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:31:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.416+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efdb00a3bb0 msgr2=0x7efdb00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.416+0000 7efdd0800640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efdb00a3bb0 0x7efdb00a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efdc8105270 tx=0x7efdbc005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.416+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 msgr2=0x7efdc819aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.416+0000 7efdd0800640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 0x7efdc819aa60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efdb8000c00 tx=0x7efdb802c650 comp rx=0 tx=0).stop 2024-04-13T03:31:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.417+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 shutdown_connections 2024-04-13T03:31:33.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.417+0000 7efdd0800640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efdb00a3bb0 0x7efdb00a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.417+0000 7efdd0800640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdc8105dd0 0x7efdc81a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.417+0000 7efdd0800640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efdc8105410 0x7efdc819aa60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.417+0000 7efdd0800640 1 --2- 172.21.15.16:0/440360245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efdc8104220 0x7efdc819a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.417+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 >> 172.21.15.16:0/440360245 conn(0x7efdc80ff7b0 msgr2=0x7efdc81086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.417+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 shutdown_connections 2024-04-13T03:31:33.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.417+0000 7efdd0800640 1 -- 172.21.15.16:0/440360245 wait complete. 2024-04-13T03:31:33.439 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-13T03:31:33.439 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:31:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.644+0000 7fcc94600640 1 -- 172.21.15.16:0/595511040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8c104200 msgr2=0x7fcc8c104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.644+0000 7fcc94600640 1 --2- 172.21.15.16:0/595511040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8c104200 0x7fcc8c104620 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009900 tx=0x7fcc8002bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.645+0000 7fcc94600640 1 -- 172.21.15.16:0/595511040 shutdown_connections 2024-04-13T03:31:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.645+0000 7fcc94600640 1 --2- 172.21.15.16:0/595511040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc8c105db0 0x7fcc8c10c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.645+0000 7fcc94600640 1 --2- 172.21.15.16:0/595511040 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8c1053f0 0x7fcc8c105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.645+0000 7fcc94600640 1 --2- 172.21.15.16:0/595511040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8c104200 0x7fcc8c104620 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.645+0000 7fcc94600640 1 -- 172.21.15.16:0/595511040 >> 172.21.15.16:0/595511040 conn(0x7fcc8c0ff7b0 msgr2=0x7fcc8c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.645+0000 7fcc94600640 1 -- 172.21.15.16:0/595511040 shutdown_connections 2024-04-13T03:31:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.645+0000 7fcc94600640 1 -- 172.21.15.16:0/595511040 wait complete. 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.646+0000 7fcc94600640 1 Processor -- start 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.646+0000 7fcc94600640 1 -- start start 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.646+0000 7fcc94600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8c104200 0x7fcc8c19a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc94600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc8c1053f0 0x7fcc8c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc94600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8c105db0 0x7fcc8c1a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc94600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c10fee0 con 0x7fcc8c105db0 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc94600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c10fd60 con 0x7fcc8c1053f0 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc94600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8c110060 con 0x7fcc8c104200 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc91200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc8c1053f0 0x7fcc8c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc91200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc8c1053f0 0x7fcc8c19aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:44596/0 (socket says 172.21.15.16:44596) 2024-04-13T03:31:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc91c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8c104200 0x7fcc8c19a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc91200640 1 -- 172.21.15.16:0/917040233 learned_addr learned my addr 172.21.15.16:0/917040233 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc92600640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8c105db0 0x7fcc8c1a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc91200640 1 -- 172.21.15.16:0/917040233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8c104200 msgr2=0x7fcc8c19a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc91200640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8c104200 0x7fcc8c19a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc91200640 1 -- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8c105db0 msgr2=0x7fcc8c1a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc91200640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8c105db0 0x7fcc8c1a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.647+0000 7fcc91200640 1 -- 172.21.15.16:0/917040233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc8c1a2180 con 0x7fcc8c1053f0 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.648+0000 7fcc91200640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc8c1053f0 0x7fcc8c19aaa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e9f0 tx=0x7fcc7800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.648+0000 7fcc8a000640 1 -- 172.21.15.16:0/917040233 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7800ccc0 con 0x7fcc8c1053f0 2024-04-13T03:31:33.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.648+0000 7fcc8a000640 1 -- 172.21.15.16:0/917040233 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc7800ce60 con 0x7fcc8c1053f0 2024-04-13T03:31:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.648+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc8c1a2470 con 0x7fcc8c1053f0 2024-04-13T03:31:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.648+0000 7fcc8a000640 1 -- 172.21.15.16:0/917040233 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc78010690 con 0x7fcc8c1053f0 2024-04-13T03:31:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.648+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc8c1a29c0 con 0x7fcc8c1053f0 2024-04-13T03:31:33.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.649+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc8c10b520 con 0x7fcc8c1053f0 2024-04-13T03:31:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.651+0000 7fcc8a000640 1 -- 172.21.15.16:0/917040233 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcc78004ea0 con 0x7fcc8c1053f0 2024-04-13T03:31:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.653+0000 7fcc8a000640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcc740a3c30 0x7fcc740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.653+0000 7fcc8a000640 1 -- 172.21.15.16:0/917040233 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcc780d0360 con 0x7fcc8c1053f0 2024-04-13T03:31:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.653+0000 7fcc91c00640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcc740a3c30 0x7fcc740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.654+0000 7fcc91c00640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcc740a3c30 0x7fcc740a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc8002c170 tx=0x7fcc8002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.657+0000 7fcc8a000640 1 -- 172.21.15.16:0/917040233 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc7809d140 con 0x7fcc8c1053f0 2024-04-13T03:31:33.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:33 smithi119 ceph-mon[35835]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:33.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:33 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2131316759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:33.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:33 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:33 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:33 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:33 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:33 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/440360245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:33.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:33 smithi016 ceph-mon[31772]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:33.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:33 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2131316759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:33.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:33 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:33 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:33 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:33 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:33 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/440360245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:33.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:33 smithi188 ceph-mon[36398]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:33.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:33 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2131316759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:33.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:33 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:33.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:33 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/440360245' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:33.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.874+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcc8c062bc0 con 0x7fcc8c1053f0 2024-04-13T03:31:33.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.875+0000 7fcc8a000640 1 -- 172.21.15.16:0/917040233 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fcc7809ca40 con 0x7fcc8c1053f0 2024-04-13T03:31:33.876 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:31:33.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcc740a3c30 msgr2=0x7fcc740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcc740a3c30 0x7fcc740a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc8002c170 tx=0x7fcc8002df90 comp rx=0 tx=0).stop 2024-04-13T03:31:33.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc8c1053f0 msgr2=0x7fcc8c19aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:33.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc8c1053f0 0x7fcc8c19aaa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800e9f0 tx=0x7fcc7800edf0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 shutdown_connections 2024-04-13T03:31:33.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcc740a3c30 0x7fcc740a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8c105db0 0x7fcc8c1a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc8c1053f0 0x7fcc8c19aaa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 --2- 172.21.15.16:0/917040233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc8c104200 0x7fcc8c19a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:33.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 >> 172.21.15.16:0/917040233 conn(0x7fcc8c0ff7b0 msgr2=0x7fcc8c1086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:33.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 shutdown_connections 2024-04-13T03:31:33.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:33.879+0000 7fcc94600640 1 -- 172.21.15.16:0/917040233 wait complete. 2024-04-13T03:31:33.901 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-13T03:31:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.098+0000 7fd321400640 1 -- 172.21.15.16:0/3349562138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c101230 msgr2=0x7fd31c101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.098+0000 7fd321400640 1 --2- 172.21.15.16:0/3349562138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c101230 0x7fd31c101690 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd308009900 tx=0x7fd30802bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.098+0000 7fd321400640 1 -- 172.21.15.16:0/3349562138 shutdown_connections 2024-04-13T03:31:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.098+0000 7fd321400640 1 --2- 172.21.15.16:0/3349562138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd31c101bd0 0x7fd31c102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.098+0000 7fd321400640 1 --2- 172.21.15.16:0/3349562138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c101230 0x7fd31c101690 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.098+0000 7fd321400640 1 --2- 172.21.15.16:0/3349562138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd31c1008d0 0x7fd31c100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.099+0000 7fd321400640 1 -- 172.21.15.16:0/3349562138 >> 172.21.15.16:0/3349562138 conn(0x7fd31c0fc470 msgr2=0x7fd31c0fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.099+0000 7fd321400640 1 -- 172.21.15.16:0/3349562138 shutdown_connections 2024-04-13T03:31:34.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.099+0000 7fd321400640 1 -- 172.21.15.16:0/3349562138 wait complete. 2024-04-13T03:31:34.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.100+0000 7fd321400640 1 Processor -- start 2024-04-13T03:31:34.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.100+0000 7fd321400640 1 -- start start 2024-04-13T03:31:34.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.100+0000 7fd321400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c1008d0 0x7fd31c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:34.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.100+0000 7fd321400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd31c101230 0x7fd31c19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:34.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd321400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd31c101bd0 0x7fd31c1a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:34.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd321400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c10fef0 con 0x7fd31c101bd0 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd321400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c10fd70 con 0x7fd31c1008d0 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd321400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c110070 con 0x7fd31c101230 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c1008d0 0x7fd31c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c1008d0 0x7fd31c19a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:44624/0 (socket says 172.21.15.16:44624) 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31a800640 1 -- 172.21.15.16:0/1442228923 learned_addr learned my addr 172.21.15.16:0/1442228923 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31b200640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd31c101bd0 0x7fd31c1a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd319e00640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd31c101230 0x7fd31c19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31a800640 1 -- 172.21.15.16:0/1442228923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd31c101230 msgr2=0x7fd31c19aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31a800640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd31c101230 0x7fd31c19aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31a800640 1 -- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd31c101bd0 msgr2=0x7fd31c1a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31a800640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd31c101bd0 0x7fd31c1a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31a800640 1 -- 172.21.15.16:0/1442228923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c1a20e0 con 0x7fd31c1008d0 2024-04-13T03:31:34.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.101+0000 7fd31b200640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd31c101bd0 0x7fd31c1a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:34.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.102+0000 7fd31a800640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c1008d0 0x7fd31c19a4c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00e9f0 tx=0x7fd30c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:34.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.102+0000 7fd313400640 1 -- 172.21.15.16:0/1442228923 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c00cd80 con 0x7fd31c1008d0 2024-04-13T03:31:34.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.102+0000 7fd313400640 1 -- 172.21.15.16:0/1442228923 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd30c010910 con 0x7fd31c1008d0 2024-04-13T03:31:34.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.102+0000 7fd313400640 1 -- 172.21.15.16:0/1442228923 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c018990 con 0x7fd31c1008d0 2024-04-13T03:31:34.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.102+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31c1a23d0 con 0x7fd31c1008d0 2024-04-13T03:31:34.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.102+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31c1a2940 con 0x7fd31c1008d0 2024-04-13T03:31:34.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.103+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd31c10b760 con 0x7fd31c1008d0 2024-04-13T03:31:34.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.105+0000 7fd313400640 1 -- 172.21.15.16:0/1442228923 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd30c010ac0 con 0x7fd31c1008d0 2024-04-13T03:31:34.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.107+0000 7fd313400640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3000a3c30 0x7fd3000a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:34.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.107+0000 7fd313400640 1 -- 172.21.15.16:0/1442228923 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd30c0d0290 con 0x7fd31c1008d0 2024-04-13T03:31:34.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.107+0000 7fd319e00640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3000a3c30 0x7fd3000a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:34.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.108+0000 7fd319e00640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3000a3c30 0x7fd3000a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd30802c240 tx=0x7fd30802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:34.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.111+0000 7fd313400640 1 -- 172.21.15.16:0/1442228923 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd30c09ca20 con 0x7fd31c1008d0 2024-04-13T03:31:34.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.344+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd31c02d5e0 con 0x7fd31c1008d0 2024-04-13T03:31:34.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.345+0000 7fd313400640 1 -- 172.21.15.16:0/1442228923 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fd30c0a0d20 con 0x7fd31c1008d0 2024-04-13T03:31:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:34.346 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-04-13T03:31:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:31:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-04-13T03:31:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-04-13T03:31:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:31:34.347 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-04-13T03:31:34.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3000a3c30 msgr2=0x7fd3000a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:34.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3000a3c30 0x7fd3000a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd30802c240 tx=0x7fd30802df90 comp rx=0 tx=0).stop 2024-04-13T03:31:34.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c1008d0 msgr2=0x7fd31c19a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:34.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c1008d0 0x7fd31c19a4c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00e9f0 tx=0x7fd30c00eeb0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 shutdown_connections 2024-04-13T03:31:34.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3000a3c30 0x7fd3000a60f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd31c101bd0 0x7fd31c1a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd31c101230 0x7fd31c19aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 --2- 172.21.15.16:0/1442228923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd31c1008d0 0x7fd31c19a4c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 >> 172.21.15.16:0/1442228923 conn(0x7fd31c0fc470 msgr2=0x7fd31c0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:34.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 shutdown_connections 2024-04-13T03:31:34.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.348+0000 7fd321400640 1 -- 172.21.15.16:0/1442228923 wait complete. 2024-04-13T03:31:34.368 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-13T03:31:34.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-13T03:31:34.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.565+0000 7fa257200640 1 -- 172.21.15.16:0/1619338392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103080 msgr2=0x7fa258103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:34.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.565+0000 7fa257200640 1 --2- 172.21.15.16:0/1619338392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103080 0x7fa258103500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa244009860 tx=0x7fa24402bb70 comp rx=0 tx=0).stop 2024-04-13T03:31:34.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.565+0000 7fa257200640 1 -- 172.21.15.16:0/1619338392 shutdown_connections 2024-04-13T03:31:34.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.565+0000 7fa257200640 1 --2- 172.21.15.16:0/1619338392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa258103a40 0x7fa25810a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.565+0000 7fa257200640 1 --2- 172.21.15.16:0/1619338392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103080 0x7fa258103500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.565+0000 7fa257200640 1 --2- 172.21.15.16:0/1619338392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa258101ea0 0x7fa2581022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.565+0000 7fa257200640 1 -- 172.21.15.16:0/1619338392 >> 172.21.15.16:0/1619338392 conn(0x7fa2580fd460 msgr2=0x7fa2580ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:34.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.565+0000 7fa257200640 1 -- 172.21.15.16:0/1619338392 shutdown_connections 2024-04-13T03:31:34.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.565+0000 7fa257200640 1 -- 172.21.15.16:0/1619338392 wait complete. 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.566+0000 7fa257200640 1 Processor -- start 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa257200640 1 -- start start 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa257200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa258101ea0 0x7fa258075d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa257200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa258103080 0x7fa2580743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa257200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103a40 0x7fa2580748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa257200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa25810db90 con 0x7fa258103080 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa257200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa25810da10 con 0x7fa258103a40 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa257200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa25810dd10 con 0x7fa258101ea0 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa256800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103a40 0x7fa2580748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa255400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa258103080 0x7fa2580743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:34.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa256800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103a40 0x7fa2580748f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:44646/0 (socket says 172.21.15.16:44646) 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa255400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa258103080 0x7fa2580743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42310/0 (socket says 172.21.15.16:42310) 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa256800640 1 -- 172.21.15.16:0/2449655569 learned_addr learned my addr 172.21.15.16:0/2449655569 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.567+0000 7fa255e00640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa258101ea0 0x7fa258075d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa256800640 1 -- 172.21.15.16:0/2449655569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa258101ea0 msgr2=0x7fa258075d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa256800640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa258101ea0 0x7fa258075d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa256800640 1 -- 172.21.15.16:0/2449655569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa258103080 msgr2=0x7fa2580743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa256800640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa258103080 0x7fa2580743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa256800640 1 -- 172.21.15.16:0/2449655569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa258074e30 con 0x7fa258103a40 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa255e00640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa258101ea0 0x7fa258075d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:31:34.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa256800640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103a40 0x7fa2580748f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa250009fd0 tx=0x7fa25000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:34.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa24ea00640 1 -- 172.21.15.16:0/2449655569 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa250004b20 con 0x7fa258103a40 2024-04-13T03:31:34.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa258075120 con 0x7fa258103a40 2024-04-13T03:31:34.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa24ea00640 1 -- 172.21.15.16:0/2449655569 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa250003e50 con 0x7fa258103a40 2024-04-13T03:31:34.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa24ea00640 1 -- 172.21.15.16:0/2449655569 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25000b630 con 0x7fa258103a40 2024-04-13T03:31:34.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.568+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2581a6c80 con 0x7fa258103a40 2024-04-13T03:31:34.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.570+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa258109290 con 0x7fa258103a40 2024-04-13T03:31:34.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.571+0000 7fa24ea00640 1 -- 172.21.15.16:0/2449655569 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa25001d020 con 0x7fa258103a40 2024-04-13T03:31:34.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.572+0000 7fa24ea00640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa23c0a3bb0 0x7fa23c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:34.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.573+0000 7fa24ea00640 1 -- 172.21.15.16:0/2449655569 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa2500cfff0 con 0x7fa258103a40 2024-04-13T03:31:34.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.573+0000 7fa255e00640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa23c0a3bb0 0x7fa23c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:34.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.574+0000 7fa255e00640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa23c0a3bb0 0x7fa23c0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa258102bf0 tx=0x7fa248005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:34.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.577+0000 7fa24ea00640 1 -- 172.21.15.16:0/2449655569 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa25009ce30 con 0x7fa258103a40 2024-04-13T03:31:34.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:34.785+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa258062bc0 con 0x7fa258103a40 2024-04-13T03:31:34.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:34 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/917040233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:31:34.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:34 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1442228923' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-13T03:31:34.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:34 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:34.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:34 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:34.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:34 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:34.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:34 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:31:34.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:34 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:34.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:34 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/917040233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:31:34.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:34 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1442228923' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-13T03:31:34.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:34 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:34.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:34 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:34.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:34 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:34.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:34 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:31:34.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:34 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:34.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:34 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/917040233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:31:34.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:34 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1442228923' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-13T03:31:34.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:34 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:34.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:34 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:34.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:34 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:34.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:34 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:31:34.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:34 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:35.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.536+0000 7fa24ea00640 1 -- 172.21.15.16:0/2449655569 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7fa25009c730 con 0x7fa258103a40 2024-04-13T03:31:35.538 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-04-13T03:31:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa23c0a3bb0 msgr2=0x7fa23c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa23c0a3bb0 0x7fa23c0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa258102bf0 tx=0x7fa248005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103a40 msgr2=0x7fa2580748f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103a40 0x7fa2580748f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa250009fd0 tx=0x7fa25000c800 comp rx=0 tx=0).stop 2024-04-13T03:31:35.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 shutdown_connections 2024-04-13T03:31:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa23c0a3bb0 0x7fa23c0a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa258103a40 0x7fa2580748f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa258103080 0x7fa2580743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 --2- 172.21.15.16:0/2449655569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa258101ea0 0x7fa258075d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.542+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 >> 172.21.15.16:0/2449655569 conn(0x7fa2580fd460 msgr2=0x7fa258106460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.543+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 shutdown_connections 2024-04-13T03:31:35.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.544+0000 7fa257200640 1 -- 172.21.15.16:0/2449655569 wait complete. 2024-04-13T03:31:35.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-13T03:31:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.795+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3833190836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9ec1031f0 msgr2=0x7fc9ec103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.795+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3833190836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9ec1031f0 0x7fc9ec103670 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009860 tx=0x7fc9d802bb70 comp rx=0 tx=0).stop 2024-04-13T03:31:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.795+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3833190836 shutdown_connections 2024-04-13T03:31:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.795+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3833190836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec103bb0 0x7fc9ec10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.795+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3833190836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9ec1031f0 0x7fc9ec103670 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:35.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.795+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3833190836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec102000 0x7fc9ec102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.795+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3833190836 >> 172.21.15.16:0/3833190836 conn(0x7fc9ec0fd570 msgr2=0x7fc9ec0ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.796+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3833190836 shutdown_connections 2024-04-13T03:31:35.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.796+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3833190836 wait complete. 2024-04-13T03:31:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.797+0000 7fc9f0e00640 1 Processor -- start 2024-04-13T03:31:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.797+0000 7fc9f0e00640 1 -- start start 2024-04-13T03:31:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.797+0000 7fc9f0e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102000 0x7fc9ec1982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:35.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.797+0000 7fc9f0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec1031f0 0x7fc9ec198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.797+0000 7fc9f0e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9ec103bb0 0x7fc9ec19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.797+0000 7fc9f0e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec10dc90 con 0x7fc9ec1031f0 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.797+0000 7fc9f0e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec10db10 con 0x7fc9ec102000 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.797+0000 7fc9f0e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec10de10 con 0x7fc9ec103bb0 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9e9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec1031f0 0x7fc9ec198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9e9400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec1031f0 0x7fc9ec198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42332/0 (socket says 172.21.15.16:42332) 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9e9400640 1 -- 172.21.15.16:0/3814267175 learned_addr learned my addr 172.21.15.16:0/3814267175 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9e9e00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102000 0x7fc9ec1982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9ea800640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9ec103bb0 0x7fc9ec19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9ea800640 1 -- 172.21.15.16:0/3814267175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102000 msgr2=0x7fc9ec1982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9ea800640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102000 0x7fc9ec1982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9ea800640 1 -- 172.21.15.16:0/3814267175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec1031f0 msgr2=0x7fc9ec198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9ea800640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec1031f0 0x7fc9ec198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.798+0000 7fc9ea800640 1 -- 172.21.15.16:0/3814267175 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ec19ff00 con 0x7fc9ec103bb0 2024-04-13T03:31:35.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.799+0000 7fc9ea800640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9ec103bb0 0x7fc9ec19f890 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009fd0 tx=0x7fc9e400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:35.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.799+0000 7fc9e2a00640 1 -- 172.21.15.16:0/3814267175 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e400ce20 con 0x7fc9ec103bb0 2024-04-13T03:31:35.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.799+0000 7fc9e2a00640 1 -- 172.21.15.16:0/3814267175 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9e4005020 con 0x7fc9ec103bb0 2024-04-13T03:31:35.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.799+0000 7fc9e2a00640 1 -- 172.21.15.16:0/3814267175 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4010720 con 0x7fc9ec103bb0 2024-04-13T03:31:35.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.799+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ec1a01f0 con 0x7fc9ec103bb0 2024-04-13T03:31:35.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.799+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ec1a0740 con 0x7fc9ec103bb0 2024-04-13T03:31:35.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.800+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9ec1093c0 con 0x7fc9ec103bb0 2024-04-13T03:31:35.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.802+0000 7fc9e2a00640 1 -- 172.21.15.16:0/3814267175 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc9e4005540 con 0x7fc9ec103bb0 2024-04-13T03:31:35.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.803+0000 7fc9e2a00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc9d00a3c30 0x7fc9d00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:35.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.803+0000 7fc9e2a00640 1 -- 172.21.15.16:0/3814267175 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7fc9e40d07f0 con 0x7fc9ec103bb0 2024-04-13T03:31:35.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.804+0000 7fc9e9e00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc9d00a3c30 0x7fc9d00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:35.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.804+0000 7fc9e9e00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc9d00a3c30 0x7fc9d00a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009530 tx=0x7fc9dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:35.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:35.807+0000 7fc9e2a00640 1 -- 172.21.15.16:0/3814267175 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9e409d460 con 0x7fc9ec103bb0 2024-04-13T03:31:35.820 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:35 smithi188 ceph-mon[36398]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:35.820 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:35 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:31:35.820 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:35 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2449655569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-13T03:31:35.821 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:35 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-13T03:31:35.821 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:35 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:35.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:35 smithi119 ceph-mon[35835]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:35.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:35 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:31:35.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:35 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2449655569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-13T03:31:35.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:35 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-13T03:31:35.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:35 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:35.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:35 smithi016 ceph-mon[31772]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:35.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:35 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:31:35.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:35 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2449655569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-13T03:31:35.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:35 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-13T03:31:35.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:35 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:36.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.017+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fc9ec062bc0 con 0x7fc9ec103bb0 2024-04-13T03:31:36.553 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:36 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-13T03:31:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.553+0000 7fc9e2a00640 1 -- 172.21.15.16:0/3814267175 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7fc9e409cd60 con 0x7fc9ec103bb0 2024-04-13T03:31:36.554 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-04-13T03:31:36.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.562+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc9d00a3c30 msgr2=0x7fc9d00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.562+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc9d00a3c30 0x7fc9d00a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009530 tx=0x7fc9dc009260 comp rx=0 tx=0).stop 2024-04-13T03:31:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.562+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9ec103bb0 msgr2=0x7fc9ec19f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.562+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9ec103bb0 0x7fc9ec19f890 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009fd0 tx=0x7fc9e400ef10 comp rx=0 tx=0).stop 2024-04-13T03:31:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.563+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 shutdown_connections 2024-04-13T03:31:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.563+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc9d00a3c30 0x7fc9d00a60f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.563+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9ec103bb0 0x7fc9ec19f890 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.563+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9ec1031f0 0x7fc9ec198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.563+0000 7fc9f0e00640 1 --2- 172.21.15.16:0/3814267175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102000 0x7fc9ec1982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:36.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.563+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 >> 172.21.15.16:0/3814267175 conn(0x7fc9ec0fd570 msgr2=0x7fc9ec106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:36.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.564+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 shutdown_connections 2024-04-13T03:31:36.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.564+0000 7fc9f0e00640 1 -- 172.21.15.16:0/3814267175 wait complete. 2024-04-13T03:31:36.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-13T03:31:36.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.804+0000 7f51ad000640 1 -- 172.21.15.16:0/1059578424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8105410 msgr2=0x7f51a8105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:36.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.804+0000 7f51ad000640 1 --2- 172.21.15.16:0/1059578424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8105410 0x7f51a8105890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5194009900 tx=0x7f519402bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:36.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.804+0000 7f51ad000640 1 -- 172.21.15.16:0/1059578424 shutdown_connections 2024-04-13T03:31:36.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.804+0000 7f51ad000640 1 --2- 172.21.15.16:0/1059578424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a8105dd0 0x7f51a810c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:36.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.804+0000 7f51ad000640 1 --2- 172.21.15.16:0/1059578424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8105410 0x7f51a8105890 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:36.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.804+0000 7f51ad000640 1 --2- 172.21.15.16:0/1059578424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51a8104220 0x7f51a8104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:36.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.804+0000 7f51ad000640 1 -- 172.21.15.16:0/1059578424 >> 172.21.15.16:0/1059578424 conn(0x7f51a80ff7b0 msgr2=0x7f51a8101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:36.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.804+0000 7f51ad000640 1 -- 172.21.15.16:0/1059578424 shutdown_connections 2024-04-13T03:31:36.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.804+0000 7f51ad000640 1 -- 172.21.15.16:0/1059578424 wait complete. 2024-04-13T03:31:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.805+0000 7f51ad000640 1 Processor -- start 2024-04-13T03:31:36.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.805+0000 7f51ad000640 1 -- start start 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51ad000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8104220 0x7f51a819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51ad000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51a8105410 0x7f51a819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51ad000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a8105dd0 0x7f51a81a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51ad000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51a810ff00 con 0x7f51a8104220 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51ad000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f51a810fd80 con 0x7f51a8105dd0 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51ad000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f51a8110080 con 0x7f51a8105410 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8104220 0x7f51a819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8104220 0x7f51a819a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42350/0 (socket says 172.21.15.16:42350) 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a5e00640 1 -- 172.21.15.16:0/4179464643 learned_addr learned my addr 172.21.15.16:0/4179464643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a5400640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51a8105410 0x7f51a819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a6800640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a8105dd0 0x7f51a81a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a5e00640 1 -- 172.21.15.16:0/4179464643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51a8105410 msgr2=0x7f51a819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:36.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a5e00640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51a8105410 0x7f51a819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a5e00640 1 -- 172.21.15.16:0/4179464643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a8105dd0 msgr2=0x7f51a81a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a5e00640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a8105dd0 0x7f51a81a1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.806+0000 7f51a5e00640 1 -- 172.21.15.16:0/4179464643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a81a2120 con 0x7f51a8104220 2024-04-13T03:31:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.807+0000 7f51a5e00640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8104220 0x7f51a819a500 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f519800e9c0 tx=0x7f519800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:36.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.807+0000 7f519ea00640 1 -- 172.21.15.16:0/4179464643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519800ccc0 con 0x7f51a8104220 2024-04-13T03:31:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.807+0000 7f519ea00640 1 -- 172.21.15.16:0/4179464643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51980035f0 con 0x7f51a8104220 2024-04-13T03:31:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.807+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a81a2410 con 0x7f51a8104220 2024-04-13T03:31:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.807+0000 7f519ea00640 1 -- 172.21.15.16:0/4179464643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519800ce60 con 0x7f51a8104220 2024-04-13T03:31:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.807+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a81a2930 con 0x7f51a8104220 2024-04-13T03:31:36.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.808+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a810b540 con 0x7f51a8104220 2024-04-13T03:31:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.811+0000 7f519ea00640 1 -- 172.21.15.16:0/4179464643 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f51980037a0 con 0x7f51a8104220 2024-04-13T03:31:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.812+0000 7f519ea00640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f518c0a3bb0 0x7f518c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.812+0000 7f519ea00640 1 -- 172.21.15.16:0/4179464643 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f51980d06f0 con 0x7f51a8104220 2024-04-13T03:31:36.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.813+0000 7f51a5400640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f518c0a3bb0 0x7f518c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:36.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.813+0000 7f51a5400640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f518c0a3bb0 0x7f518c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f519402c240 tx=0x7f519402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:36.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:36.815+0000 7f519ea00640 1 -- 172.21.15.16:0/4179464643 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f519809d1e0 con 0x7f51a8104220 2024-04-13T03:31:36.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:36 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-13T03:31:36.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:36 smithi119 ceph-mon[35835]: osdmap e83: 12 total, 12 up, 12 in 2024-04-13T03:31:36.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:36 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3814267175' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-13T03:31:36.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:36 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-13T03:31:36.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:36.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:36.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:36 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:36 smithi016 ceph-mon[31772]: osdmap e83: 12 total, 12 up, 12 in 2024-04-13T03:31:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:36 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3814267175' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-13T03:31:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:36 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-13T03:31:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:36 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:36.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:36 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-13T03:31:36.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:36 smithi188 ceph-mon[36398]: osdmap e83: 12 total, 12 up, 12 in 2024-04-13T03:31:36.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:36 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3814267175' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-13T03:31:36.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:36 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-13T03:31:36.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:36.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:36.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:36 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:37.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.107+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f51a802d5e0 con 0x7f51a8104220 2024-04-13T03:31:37.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.549+0000 7f519ea00640 1 -- 172.21.15.16:0/4179464643 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f519809cae0 con 0x7f51a8104220 2024-04-13T03:31:37.551 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2024-04-13T03:31:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f518c0a3bb0 msgr2=0x7f518c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f518c0a3bb0 0x7f518c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f519402c240 tx=0x7f519402df90 comp rx=0 tx=0).stop 2024-04-13T03:31:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8104220 msgr2=0x7f51a819a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8104220 0x7f51a819a500 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f519800e9c0 tx=0x7f519800edc0 comp rx=0 tx=0).stop 2024-04-13T03:31:37.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 shutdown_connections 2024-04-13T03:31:37.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f518c0a3bb0 0x7f518c0a6070 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f519402c240 tx=0x7f519402df90 comp rx=0 tx=0).stop 2024-04-13T03:31:37.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a8105dd0 0x7f51a81a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:37.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51a8105410 0x7f51a819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:37.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 --2- 172.21.15.16:0/4179464643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51a8104220 0x7f51a819a500 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:37.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.553+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 >> 172.21.15.16:0/4179464643 conn(0x7f51a80ff7b0 msgr2=0x7f51a8108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:37.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.554+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 shutdown_connections 2024-04-13T03:31:37.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.555+0000 7f51ad000640 1 -- 172.21.15.16:0/4179464643 wait complete. 2024-04-13T03:31:37.608 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-13T03:31:37.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-13T03:31:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.804+0000 7f3936800640 1 -- 172.21.15.16:0/2764185022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 msgr2=0x7f3930104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.804+0000 7f3936800640 1 --2- 172.21.15.16:0/2764185022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 0x7f3930104e80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3920009860 tx=0x7f392002bb70 comp rx=0 tx=0).stop 2024-04-13T03:31:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.804+0000 7f3936800640 1 -- 172.21.15.16:0/2764185022 shutdown_connections 2024-04-13T03:31:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.804+0000 7f3936800640 1 --2- 172.21.15.16:0/2764185022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3930107cf0 0x7f393010a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.804+0000 7f3936800640 1 --2- 172.21.15.16:0/2764185022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39301053c0 0x7f39301077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:37.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.804+0000 7f3936800640 1 --2- 172.21.15.16:0/2764185022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 0x7f3930104e80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.804+0000 7f3936800640 1 -- 172.21.15.16:0/2764185022 >> 172.21.15.16:0/2764185022 conn(0x7f39300fc480 msgr2=0x7f39300fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.804+0000 7f3936800640 1 -- 172.21.15.16:0/2764185022 shutdown_connections 2024-04-13T03:31:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.804+0000 7f3936800640 1 -- 172.21.15.16:0/2764185022 wait complete. 2024-04-13T03:31:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.805+0000 7f3936800640 1 Processor -- start 2024-04-13T03:31:37.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f3936800640 1 -- start start 2024-04-13T03:31:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f3936800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 0x7f3930102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f3936800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39301053c0 0x7f3930100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f3936800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3930107cf0 0x7f3930100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f3936800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f393010dca0 con 0x7f39301053c0 2024-04-13T03:31:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f3936800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f393010db20 con 0x7f3930107cf0 2024-04-13T03:31:37.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f3936800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f393010de20 con 0x7f3930102a90 2024-04-13T03:31:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f392f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 0x7f3930102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f392f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 0x7f3930102400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:39508/0 (socket says 172.21.15.16:39508) 2024-04-13T03:31:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.806+0000 7f392f400640 1 -- 172.21.15.16:0/295634502 learned_addr learned my addr 172.21.15.16:0/295634502 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.807+0000 7f3927e00640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39301053c0 0x7f3930100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.807+0000 7f392fe00640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3930107cf0 0x7f3930100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.807+0000 7f392f400640 1 -- 172.21.15.16:0/295634502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3930107cf0 msgr2=0x7f3930100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.807+0000 7f392f400640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3930107cf0 0x7f3930100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:37.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.807+0000 7f392f400640 1 -- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39301053c0 msgr2=0x7f3930100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.807+0000 7f392f400640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39301053c0 0x7f3930100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.807+0000 7f392f400640 1 -- 172.21.15.16:0/295634502 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3930101810 con 0x7f3930102a90 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.807+0000 7f392fe00640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3930107cf0 0x7f3930100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.807+0000 7f392f400640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 0x7f3930102400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3920037040 tx=0x7f39200323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.808+0000 7f392cc00640 1 -- 172.21.15.16:0/295634502 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920041030 con 0x7f3930102a90 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.808+0000 7f392cc00640 1 -- 172.21.15.16:0/295634502 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f392002c510 con 0x7f3930102a90 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.808+0000 7f392cc00640 1 -- 172.21.15.16:0/295634502 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920039040 con 0x7f3930102a90 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.808+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39301a2420 con 0x7f3930102a90 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.808+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39301a29c0 con 0x7f3930102a90 2024-04-13T03:31:37.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.809+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39300fdf70 con 0x7f3930102a90 2024-04-13T03:31:37.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.811+0000 7f392cc00640 1 -- 172.21.15.16:0/295634502 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3920044070 con 0x7f3930102a90 2024-04-13T03:31:37.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.812+0000 7f392cc00640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39140a3c30 0x7f39140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:37.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.812+0000 7f392cc00640 1 -- 172.21.15.16:0/295634502 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f39200f1810 con 0x7f3930102a90 2024-04-13T03:31:37.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.813+0000 7f3927e00640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39140a3c30 0x7f39140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:37.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.813+0000 7f3927e00640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39140a3c30 0x7f39140a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3930102170 tx=0x7f3918009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:37.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:37.817+0000 7f392cc00640 1 -- 172.21.15.16:0/295634502 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39200be2a0 con 0x7f3930102a90 2024-04-13T03:31:37.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:37 smithi119 ceph-mon[35835]: pgmap v311: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:37.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:37 smithi119 ceph-mon[35835]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-13T03:31:37.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:37 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:37.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:37 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-13T03:31:37.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:37 smithi119 ceph-mon[35835]: osdmap e84: 12 total, 12 up, 12 in 2024-04-13T03:31:37.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:37 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4179464643' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-13T03:31:37.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:37 smithi119 ceph-mon[35835]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-13T03:31:37.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:37 smithi016 ceph-mon[31772]: pgmap v311: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:37.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:37 smithi016 ceph-mon[31772]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-13T03:31:37.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:37 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:37.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:37 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-13T03:31:37.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:37 smithi016 ceph-mon[31772]: osdmap e84: 12 total, 12 up, 12 in 2024-04-13T03:31:37.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:37 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4179464643' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-13T03:31:37.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:37 smithi016 ceph-mon[31772]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-13T03:31:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:37 smithi188 ceph-mon[36398]: pgmap v311: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:37 smithi188 ceph-mon[36398]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-13T03:31:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:37 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:37 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-13T03:31:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:37 smithi188 ceph-mon[36398]: osdmap e84: 12 total, 12 up, 12 in 2024-04-13T03:31:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:37 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4179464643' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-13T03:31:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:37 smithi188 ceph-mon[36398]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-13T03:31:38.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.032+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f393002d5e0 con 0x7f3930102a90 2024-04-13T03:31:38.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.572+0000 7f392cc00640 1 -- 172.21.15.16:0/295634502 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7f39200bdba0 con 0x7f3930102a90 2024-04-13T03:31:38.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39140a3c30 msgr2=0x7f39140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39140a3c30 0x7f39140a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3930102170 tx=0x7f3918009260 comp rx=0 tx=0).stop 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 msgr2=0x7f3930102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 0x7f3930102400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3920037040 tx=0x7f39200323d0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 shutdown_connections 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39140a3c30 0x7f39140a60f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3930107cf0 0x7f3930100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39301053c0 0x7f3930100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 --2- 172.21.15.16:0/295634502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3930102a90 0x7f3930102400 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.576+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 >> 172.21.15.16:0/295634502 conn(0x7f39300fc480 msgr2=0x7f3930103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.577+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 shutdown_connections 2024-04-13T03:31:38.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.577+0000 7f3936800640 1 -- 172.21.15.16:0/295634502 wait complete. 2024-04-13T03:31:38.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-13T03:31:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.827+0000 7f4fc3e00640 1 -- 172.21.15.16:0/2285791627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4106490 msgr2=0x7f4fc4108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:38.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.827+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/2285791627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4106490 0x7f4fc4108880 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc009ca0 tx=0x7f4fbc02c0f0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.827+0000 7f4fc3e00640 1 -- 172.21.15.16:0/2285791627 shutdown_connections 2024-04-13T03:31:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.827+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/2285791627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4106490 0x7f4fc4108880 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.827+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/2285791627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc4101790 0x7f4fc4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.827+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/2285791627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc4068470 0x7f4fc4101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.827+0000 7f4fc3e00640 1 -- 172.21.15.16:0/2285791627 >> 172.21.15.16:0/2285791627 conn(0x7f4fc40fb5f0 msgr2=0x7f4fc40fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.827+0000 7f4fc3e00640 1 -- 172.21.15.16:0/2285791627 shutdown_connections 2024-04-13T03:31:38.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.827+0000 7f4fc3e00640 1 -- 172.21.15.16:0/2285791627 wait complete. 2024-04-13T03:31:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.828+0000 7f4fc3e00640 1 Processor -- start 2024-04-13T03:31:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.828+0000 7f4fc3e00640 1 -- start start 2024-04-13T03:31:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.828+0000 7f4fc3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4068470 0x7f4fc419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.828+0000 7f4fc3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc4101790 0x7f4fc419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc4106490 0x7f4fc41a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:38.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc3e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc410daf0 con 0x7f4fc4068470 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc3e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc410d970 con 0x7f4fc4106490 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc410dc70 con 0x7f4fc4101790 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4068470 0x7f4fc419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4068470 0x7f4fc419a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42380/0 (socket says 172.21.15.16:42380) 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc2a00640 1 -- 172.21.15.16:0/598790533 learned_addr learned my addr 172.21.15.16:0/598790533 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc3400640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc4106490 0x7f4fc41a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc2000640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc4101790 0x7f4fc419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc2a00640 1 -- 172.21.15.16:0/598790533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc4101790 msgr2=0x7f4fc419a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc2a00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc4101790 0x7f4fc419a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc2a00640 1 -- 172.21.15.16:0/598790533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc4106490 msgr2=0x7f4fc41a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc2a00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc4106490 0x7f4fc41a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc2a00640 1 -- 172.21.15.16:0/598790533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc41a1f40 con 0x7f4fc4068470 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.829+0000 7f4fc3400640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc4106490 0x7f4fc41a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.830+0000 7f4fc2a00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4068470 0x7f4fc419a320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400e9c0 tx=0x7f4fb400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:38.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.830+0000 7f4fc8a00640 1 -- 172.21.15.16:0/598790533 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb400cd80 con 0x7f4fc4068470 2024-04-13T03:31:38.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.830+0000 7f4fc8a00640 1 -- 172.21.15.16:0/598790533 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fb40035f0 con 0x7f4fc4068470 2024-04-13T03:31:38.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.830+0000 7f4fc8a00640 1 -- 172.21.15.16:0/598790533 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4010690 con 0x7f4fc4068470 2024-04-13T03:31:38.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.830+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc41a2230 con 0x7f4fc4068470 2024-04-13T03:31:38.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.830+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc41a2730 con 0x7f4fc4068470 2024-04-13T03:31:38.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.831+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc41a2a50 con 0x7f4fc4068470 2024-04-13T03:31:38.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.834+0000 7f4fc8a00640 1 -- 172.21.15.16:0/598790533 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4fb40037a0 con 0x7f4fc4068470 2024-04-13T03:31:38.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.835+0000 7f4fc8a00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fa80a3c30 0x7f4fa80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.836+0000 7f4fc8a00640 1 -- 172.21.15.16:0/598790533 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7f4fb40d0780 con 0x7f4fc4068470 2024-04-13T03:31:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.836+0000 7f4fc2000640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fa80a3c30 0x7f4fa80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:38.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.837+0000 7f4fc2000640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fa80a3c30 0x7f4fa80a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00095f0 tx=0x7f4fb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:38.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:38.840+0000 7f4fc8a00640 1 -- 172.21.15.16:0/598790533 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4fb409d0d0 con 0x7f4fc4068470 2024-04-13T03:31:38.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4179464643' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-13T03:31:38.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: osdmap e85: 12 total, 12 up, 12 in 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] up:active 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/295634502' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-13T03:31:38.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:38 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4179464643' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: osdmap e85: 12 total, 12 up, 12 in 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] up:active 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:31:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/295634502' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-13T03:31:38.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:38 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-13T03:31:39.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:39.047+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4fc41a2d70 con 0x7f4fc4068470 2024-04-13T03:31:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4179464643' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-13T03:31:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: osdmap e85: 12 total, 12 up, 12 in 2024-04-13T03:31:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-13T03:31:39.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-13T03:31:39.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:39.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:39.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:39.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:31:39.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:31:39.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] up:active 2024-04-13T03:31:39.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:31:39.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/295634502' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-13T03:31:39.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:38 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-13T03:31:39.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:39 smithi016 ceph-mon[31772]: pgmap v314: 97 pgs: 7 active+clean, 24 creating+peering, 66 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:39.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:39 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-13T03:31:39.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:39 smithi016 ceph-mon[31772]: osdmap e86: 12 total, 12 up, 12 in 2024-04-13T03:31:39.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:39 smithi016 ceph-mon[31772]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-13T03:31:39.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:39 smithi016 ceph-mon[31772]: Cluster is now healthy 2024-04-13T03:31:39.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:39 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/598790533' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-13T03:31:39.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:39 smithi188 ceph-mon[36398]: pgmap v314: 97 pgs: 7 active+clean, 24 creating+peering, 66 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:39.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:39 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-13T03:31:39.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:39 smithi188 ceph-mon[36398]: osdmap e86: 12 total, 12 up, 12 in 2024-04-13T03:31:39.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:39 smithi188 ceph-mon[36398]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-13T03:31:39.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:39 smithi188 ceph-mon[36398]: Cluster is now healthy 2024-04-13T03:31:39.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:39 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/598790533' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-13T03:31:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:39 smithi119 ceph-mon[35835]: pgmap v314: 97 pgs: 7 active+clean, 24 creating+peering, 66 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:39 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-13T03:31:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:39 smithi119 ceph-mon[35835]: osdmap e86: 12 total, 12 up, 12 in 2024-04-13T03:31:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:39 smithi119 ceph-mon[35835]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-13T03:31:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:39 smithi119 ceph-mon[35835]: Cluster is now healthy 2024-04-13T03:31:40.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:39 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/598790533' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-13T03:31:40.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.802+0000 7f4fc8a00640 1 -- 172.21.15.16:0/598790533 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7f4fb409c9d0 con 0x7f4fc4068470 2024-04-13T03:31:40.804 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' created 2024-04-13T03:31:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.809+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fa80a3c30 msgr2=0x7f4fa80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.809+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fa80a3c30 0x7f4fa80a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00095f0 tx=0x7f4fb0005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:40.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.809+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4068470 msgr2=0x7f4fc419a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.809+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4068470 0x7f4fc419a320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400e9c0 tx=0x7f4fb400ee80 comp rx=0 tx=0).stop 2024-04-13T03:31:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.810+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 shutdown_connections 2024-04-13T03:31:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.810+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4fa80a3c30 0x7f4fa80a60f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.810+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fc4106490 0x7f4fc41a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.810+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc4101790 0x7f4fc419a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.810+0000 7f4fc3e00640 1 --2- 172.21.15.16:0/598790533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4fc4068470 0x7f4fc419a320 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:40.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.810+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 >> 172.21.15.16:0/598790533 conn(0x7f4fc40fb5f0 msgr2=0x7f4fc40fb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:40.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.811+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 shutdown_connections 2024-04-13T03:31:40.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:40.811+0000 7f4fc3e00640 1 -- 172.21.15.16:0/598790533 wait complete. 2024-04-13T03:31:40.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-13T03:31:41.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.040+0000 7f2c2de00640 1 -- 172.21.15.16:0/2110044976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c281096f0 msgr2=0x7f2c28109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.040+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2110044976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c281096f0 0x7f2c28109b70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009900 tx=0x7f2c1402bc50 comp rx=0 tx=0).stop 2024-04-13T03:31:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.040+0000 7f2c2de00640 1 -- 172.21.15.16:0/2110044976 shutdown_connections 2024-04-13T03:31:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.040+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2110044976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c2810a0b0 0x7f2c28110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.040+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2110044976 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c281096f0 0x7f2c28109b70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.040+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2110044976 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c28108500 0x7f2c28108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.040+0000 7f2c2de00640 1 -- 172.21.15.16:0/2110044976 >> 172.21.15.16:0/2110044976 conn(0x7f2c28077490 msgr2=0x7f2c280798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.040+0000 7f2c2de00640 1 -- 172.21.15.16:0/2110044976 shutdown_connections 2024-04-13T03:31:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.040+0000 7f2c2de00640 1 -- 172.21.15.16:0/2110044976 wait complete. 2024-04-13T03:31:41.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.041+0000 7f2c2de00640 1 Processor -- start 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2de00640 1 -- start start 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2de00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c28108500 0x7f2c2819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c281096f0 0x7f2c2819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2de00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c2810a0b0 0x7f2c281a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2de00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2c28114190 con 0x7f2c28108500 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2de00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2c28114010 con 0x7f2c281096f0 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2de00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2c28114310 con 0x7f2c2810a0b0 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c28108500 0x7f2c2819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c28108500 0x7f2c2819e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53514/0 (socket says 172.21.15.16:53514) 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c2ca00640 1 -- 172.21.15.16:0/2320667887 learned_addr learned my addr 172.21.15.16:0/2320667887 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.042+0000 7f2c27e00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c281096f0 0x7f2c2819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.043+0000 7f2c2d400640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c2810a0b0 0x7f2c281a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.043+0000 7f2c2ca00640 1 -- 172.21.15.16:0/2320667887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c2810a0b0 msgr2=0x7f2c281a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.043+0000 7f2c2ca00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c2810a0b0 0x7f2c281a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.043+0000 7f2c2ca00640 1 -- 172.21.15.16:0/2320667887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c281096f0 msgr2=0x7f2c2819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.043+0000 7f2c2ca00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c281096f0 0x7f2c2819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.043+0000 7f2c2ca00640 1 -- 172.21.15.16:0/2320667887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c281a6480 con 0x7f2c28108500 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.043+0000 7f2c2ca00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c28108500 0x7f2c2819e860 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00e9c0 tx=0x7f2c1c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.043+0000 7f2c25600640 1 -- 172.21.15.16:0/2320667887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c00cd80 con 0x7f2c28108500 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.043+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c281a6770 con 0x7f2c28108500 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.044+0000 7f2c25600640 1 -- 172.21.15.16:0/2320667887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c1c010910 con 0x7f2c28108500 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.044+0000 7f2c25600640 1 -- 172.21.15.16:0/2320667887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c0049c0 con 0x7f2c28108500 2024-04-13T03:31:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.044+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c281a6ce0 con 0x7f2c28108500 2024-04-13T03:31:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.045+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c2810f820 con 0x7f2c28108500 2024-04-13T03:31:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.048+0000 7f2c25600640 1 -- 172.21.15.16:0/2320667887 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2c1c01f020 con 0x7f2c28108500 2024-04-13T03:31:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.050+0000 7f2c25600640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c100a3bb0 0x7f2c100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.050+0000 7f2c25600640 1 -- 172.21.15.16:0/2320667887 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f2c1c0d0ea0 con 0x7f2c28108500 2024-04-13T03:31:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.050+0000 7f2c27e00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c100a3bb0 0x7f2c100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.051+0000 7f2c27e00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c100a3bb0 0x7f2c100a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2c1402c7d0 tx=0x7f2c14005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.053+0000 7f2c25600640 1 -- 172.21.15.16:0/2320667887 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c1c026030 con 0x7f2c28108500 2024-04-13T03:31:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:40 smithi016 ceph-mon[31772]: osdmap e87: 12 total, 12 up, 12 in 2024-04-13T03:31:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:40 smithi016 ceph-mon[31772]: pgmap v317: 97 pgs: 55 active+clean, 24 creating+peering, 18 unknown; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-13T03:31:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:40 smithi188 ceph-mon[36398]: osdmap e87: 12 total, 12 up, 12 in 2024-04-13T03:31:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:40 smithi188 ceph-mon[36398]: pgmap v317: 97 pgs: 55 active+clean, 24 creating+peering, 18 unknown; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-13T03:31:41.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.261+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f2c2802d5e0 con 0x7f2c28108500 2024-04-13T03:31:41.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:40 smithi119 ceph-mon[35835]: osdmap e87: 12 total, 12 up, 12 in 2024-04-13T03:31:41.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:40 smithi119 ceph-mon[35835]: pgmap v317: 97 pgs: 55 active+clean, 24 creating+peering, 18 unknown; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-13T03:31:41.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.806+0000 7f2c25600640 1 -- 172.21.15.16:0/2320667887 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7f2c1c09d660 con 0x7f2c28108500 2024-04-13T03:31:41.807 INFO:teuthology.orchestra.run.smithi016.stderr:set pool 4 allow_ec_overwrites to true 2024-04-13T03:31:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.810+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c100a3bb0 msgr2=0x7f2c100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.810+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c100a3bb0 0x7f2c100a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2c1402c7d0 tx=0x7f2c14005fd0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.810+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c28108500 msgr2=0x7f2c2819e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.810+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c28108500 0x7f2c2819e860 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00e9c0 tx=0x7f2c1c00edc0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.811+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 shutdown_connections 2024-04-13T03:31:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.811+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2c100a3bb0 0x7f2c100a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.811+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c2810a0b0 0x7f2c281a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.811+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c281096f0 0x7f2c2819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.811+0000 7f2c2de00640 1 --2- 172.21.15.16:0/2320667887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c28108500 0x7f2c2819e860 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:41.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.811+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 >> 172.21.15.16:0/2320667887 conn(0x7f2c28077490 msgr2=0x7f2c2810c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:41.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.812+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 shutdown_connections 2024-04-13T03:31:41.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:41.813+0000 7f2c2de00640 1 -- 172.21.15.16:0/2320667887 wait complete. 2024-04-13T03:31:41.853 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-13T03:31:42.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.043+0000 7f4c13000640 1 -- 172.21.15.16:0/3836775857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 msgr2=0x7f4c0c102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.043+0000 7f4c13000640 1 --2- 172.21.15.16:0/3836775857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 0x7f4c0c102230 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4c00009900 tx=0x7f4c0002bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.043+0000 7f4c13000640 1 -- 172.21.15.16:0/3836775857 shutdown_connections 2024-04-13T03:31:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.043+0000 7f4c13000640 1 --2- 172.21.15.16:0/3836775857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c0c103920 0x7f4c0c10a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.043+0000 7f4c13000640 1 --2- 172.21.15.16:0/3836775857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0c102f60 0x7f4c0c1033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.043+0000 7f4c13000640 1 --2- 172.21.15.16:0/3836775857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 0x7f4c0c102230 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.043+0000 7f4c13000640 1 -- 172.21.15.16:0/3836775857 >> 172.21.15.16:0/3836775857 conn(0x7f4c0c0fd380 msgr2=0x7f4c0c0ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.043+0000 7f4c13000640 1 -- 172.21.15.16:0/3836775857 shutdown_connections 2024-04-13T03:31:42.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.043+0000 7f4c13000640 1 -- 172.21.15.16:0/3836775857 wait complete. 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.044+0000 7f4c13000640 1 Processor -- start 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.044+0000 7f4c13000640 1 -- start start 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c13000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 0x7f4c0c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c13000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c0c102f60 0x7f4c0c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c13000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0c103920 0x7f4c0c1a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c13000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0c10dab0 con 0x7f4c0c101e10 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c13000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0c10d930 con 0x7f4c0c102f60 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c13000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0c10dc30 con 0x7f4c0c103920 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c11c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 0x7f4c0c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c11c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 0x7f4c0c19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53530/0 (socket says 172.21.15.16:53530) 2024-04-13T03:31:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c11c00640 1 -- 172.21.15.16:0/2854805657 learned_addr learned my addr 172.21.15.16:0/2854805657 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c11200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c0c102f60 0x7f4c0c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.045+0000 7f4c12600640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0c103920 0x7f4c0c1a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c11c00640 1 -- 172.21.15.16:0/2854805657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0c103920 msgr2=0x7f4c0c1a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c11c00640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0c103920 0x7f4c0c1a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c11c00640 1 -- 172.21.15.16:0/2854805657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c0c102f60 msgr2=0x7f4c0c19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c11c00640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c0c102f60 0x7f4c0c19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c11c00640 1 -- 172.21.15.16:0/2854805657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0c1a1f00 con 0x7f4c0c101e10 2024-04-13T03:31:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c11200640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c0c102f60 0x7f4c0c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:42.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c11c00640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 0x7f4c0c19a2e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4c00009230 tx=0x7f4c0002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c0a000640 1 -- 172.21.15.16:0/2854805657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00041070 con 0x7f4c0c101e10 2024-04-13T03:31:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0c1a2190 con 0x7f4c0c101e10 2024-04-13T03:31:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c0a000640 1 -- 172.21.15.16:0/2854805657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c00004450 con 0x7f4c0c101e10 2024-04-13T03:31:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.046+0000 7f4c0a000640 1 -- 172.21.15.16:0/2854805657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00003e20 con 0x7f4c0c101e10 2024-04-13T03:31:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.047+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c0c1a2680 con 0x7f4c0c101e10 2024-04-13T03:31:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.048+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c0c1a28e0 con 0x7f4c0c101e10 2024-04-13T03:31:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.050+0000 7f4c0a000640 1 -- 172.21.15.16:0/2854805657 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4c00044070 con 0x7f4c0c101e10 2024-04-13T03:31:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.051+0000 7f4c0a000640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4bf40a3bb0 0x7f4bf40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.051+0000 7f4c0a000640 1 -- 172.21.15.16:0/2854805657 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f4c000f21d0 con 0x7f4c0c101e10 2024-04-13T03:31:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.052+0000 7f4c11200640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4bf40a3bb0 0x7f4bf40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:42.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.052+0000 7f4c11200640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4bf40a3bb0 0x7f4bf40a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c19b7f0 tx=0x7f4bf8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.055+0000 7f4c0a000640 1 -- 172.21.15.16:0/2854805657 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c000be2e0 con 0x7f4c0c101e10 2024-04-13T03:31:42.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:41 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/598790533' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-13T03:31:42.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:41 smithi016 ceph-mon[31772]: osdmap e88: 12 total, 12 up, 12 in 2024-04-13T03:31:42.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:41 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2320667887' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-13T03:31:42.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:41 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2320667887' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-13T03:31:42.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:41 smithi016 ceph-mon[31772]: osdmap e89: 12 total, 12 up, 12 in 2024-04-13T03:31:42.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:41 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/598790533' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-13T03:31:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:41 smithi188 ceph-mon[36398]: osdmap e88: 12 total, 12 up, 12 in 2024-04-13T03:31:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:41 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2320667887' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-13T03:31:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:41 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2320667887' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-13T03:31:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:41 smithi188 ceph-mon[36398]: osdmap e89: 12 total, 12 up, 12 in 2024-04-13T03:31:42.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.301+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f4c0c1a2ab0 con 0x7f4c0c101e10 2024-04-13T03:31:42.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:41 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/598790533' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-13T03:31:42.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:41 smithi119 ceph-mon[35835]: osdmap e88: 12 total, 12 up, 12 in 2024-04-13T03:31:42.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:41 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2320667887' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-13T03:31:42.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:41 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2320667887' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-13T03:31:42.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:41 smithi119 ceph-mon[35835]: osdmap e89: 12 total, 12 up, 12 in 2024-04-13T03:31:42.821 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:31:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: 2024-04-13T03:31:42.511+0000 7f7758253700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-13T03:31:42.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.838+0000 7f4c0a000640 1 -- 172.21.15.16:0/2854805657 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7f4c000c25e0 con 0x7f4c0c101e10 2024-04-13T03:31:42.839 INFO:teuthology.orchestra.run.smithi016.stderr:added data pool 4 to fsmap 2024-04-13T03:31:42.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.841+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4bf40a3bb0 msgr2=0x7f4bf40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:42.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.841+0000 7f4c13000640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4bf40a3bb0 0x7f4bf40a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c19b7f0 tx=0x7f4bf8005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:42.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.841+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 msgr2=0x7f4c0c19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:42.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.841+0000 7f4c13000640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 0x7f4c0c19a2e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4c00009230 tx=0x7f4c0002c650 comp rx=0 tx=0).stop 2024-04-13T03:31:42.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.841+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 shutdown_connections 2024-04-13T03:31:42.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.841+0000 7f4c13000640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4bf40a3bb0 0x7f4bf40a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:42.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.841+0000 7f4c13000640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c0c103920 0x7f4c0c1a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:42.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.842+0000 7f4c13000640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c0c102f60 0x7f4c0c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:42.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.842+0000 7f4c13000640 1 --2- 172.21.15.16:0/2854805657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c0c101e10 0x7f4c0c19a2e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:42.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.842+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 >> 172.21.15.16:0/2854805657 conn(0x7f4c0c0fd380 msgr2=0x7f4c0c0741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:42.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.842+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 shutdown_connections 2024-04-13T03:31:42.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:42.842+0000 7f4c13000640 1 -- 172.21.15.16:0/2854805657 wait complete. 2024-04-13T03:31:42.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.057+0000 7f59d9000640 1 -- 172.21.15.16:0/3912944595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 msgr2=0x7f59d410c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.057+0000 7f59d9000640 1 --2- 172.21.15.16:0/3912944595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 0x7f59d410c680 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009be0 tx=0x7f59cc02bef0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.057+0000 7f59d9000640 1 -- 172.21.15.16:0/3912944595 shutdown_connections 2024-04-13T03:31:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.057+0000 7f59d9000640 1 --2- 172.21.15.16:0/3912944595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 0x7f59d410c680 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.057+0000 7f59d9000640 1 --2- 172.21.15.16:0/3912944595 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4105430 0x7f59d41058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.057+0000 7f59d9000640 1 --2- 172.21.15.16:0/3912944595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d4104240 0x7f59d4104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.057+0000 7f59d9000640 1 -- 172.21.15.16:0/3912944595 >> 172.21.15.16:0/3912944595 conn(0x7f59d40ff7b0 msgr2=0x7f59d4101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.057+0000 7f59d9000640 1 -- 172.21.15.16:0/3912944595 shutdown_connections 2024-04-13T03:31:43.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.057+0000 7f59d9000640 1 -- 172.21.15.16:0/3912944595 wait complete. 2024-04-13T03:31:43.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.058+0000 7f59d9000640 1 Processor -- start 2024-04-13T03:31:43.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.058+0000 7f59d9000640 1 -- start start 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d9000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d4104240 0x7f59d419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4105430 0x7f59d419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d9000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 0x7f59d41a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d9000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f59d410ff20 con 0x7f59d4105430 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d9000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f59d410fda0 con 0x7f59d4105df0 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d9000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f59d41100a0 con 0x7f59d4104240 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4105430 0x7f59d419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d2800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 0x7f59d41a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d2800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 0x7f59d41a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:54056/0 (socket says 172.21.15.16:54056) 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d2800640 1 -- 172.21.15.16:0/840624373 learned_addr learned my addr 172.21.15.16:0/840624373 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:43.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.059+0000 7f59d1e00640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d4104240 0x7f59d419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59d2800640 1 -- 172.21.15.16:0/840624373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d4104240 msgr2=0x7f59d419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59d2800640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d4104240 0x7f59d419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59d2800640 1 -- 172.21.15.16:0/840624373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4105430 msgr2=0x7f59d419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59d2800640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4105430 0x7f59d419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59d2800640 1 -- 172.21.15.16:0/840624373 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d41a20b0 con 0x7f59d4105df0 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59d1e00640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d4104240 0x7f59d419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59d2800640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 0x7f59d41a1a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f59cc036040 tx=0x7f59cc0337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59caa00640 1 -- 172.21.15.16:0/840624373 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc043020 con 0x7f59d4105df0 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59caa00640 1 -- 172.21.15.16:0/840624373 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59cc02ca60 con 0x7f59d4105df0 2024-04-13T03:31:43.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d41a2340 con 0x7f59d4105df0 2024-04-13T03:31:43.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59caa00640 1 -- 172.21.15.16:0/840624373 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc034690 con 0x7f59d4105df0 2024-04-13T03:31:43.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.060+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d41a28e0 con 0x7f59d4105df0 2024-04-13T03:31:43.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.061+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d410b500 con 0x7f59d4105df0 2024-04-13T03:31:43.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.063+0000 7f59caa00640 1 -- 172.21.15.16:0/840624373 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f59cc046020 con 0x7f59d4105df0 2024-04-13T03:31:43.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.064+0000 7f59caa00640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f59b80a3c30 0x7f59b80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:43.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.065+0000 7f59caa00640 1 -- 172.21.15.16:0/840624373 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f59cc0f2cc0 con 0x7f59d4105df0 2024-04-13T03:31:43.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.065+0000 7f59d1e00640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f59b80a3c30 0x7f59b80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:43.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.065+0000 7f59d1e00640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f59b80a3c30 0x7f59b80a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f59d4105290 tx=0x7f59c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:43.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.069+0000 7f59caa00640 1 -- 172.21.15.16:0/840624373 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59cc0f9080 con 0x7f59d4105df0 2024-04-13T03:31:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:42 smithi016 ceph-mon[31772]: pgmap v320: 129 pgs: 89 active+clean, 24 creating+peering, 16 unknown; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-04-13T03:31:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:42 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2854805657' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-13T03:31:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:42 smithi188 ceph-mon[36398]: pgmap v320: 129 pgs: 89 active+clean, 24 creating+peering, 16 unknown; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-04-13T03:31:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:42 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2854805657' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-13T03:31:43.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.318+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59d4062bc0 con 0x7f59d4105df0 2024-04-13T03:31:43.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:42 smithi119 ceph-mon[35835]: pgmap v320: 129 pgs: 89 active+clean, 24 creating+peering, 16 unknown; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-04-13T03:31:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:42 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2854805657' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-13T03:31:43.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.320+0000 7f59caa00640 1 -- 172.21.15.16:0/840624373 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11797 (secure 0 0 0) 0x7f59cc04d020 con 0x7f59d4105df0 2024-04-13T03:31:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:43.322 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:31:42.303966+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-13T03:31:43.322 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-13T03:31:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f59b80a3c30 msgr2=0x7f59b80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f59b80a3c30 0x7f59b80a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f59d4105290 tx=0x7f59c4009260 comp rx=0 tx=0).stop 2024-04-13T03:31:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 msgr2=0x7f59d41a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 0x7f59d41a1a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f59cc036040 tx=0x7f59cc0337b0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 shutdown_connections 2024-04-13T03:31:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f59b80a3c30 0x7f59b80a60f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59d4105df0 0x7f59d41a1a40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59d4105430 0x7f59d419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 --2- 172.21.15.16:0/840624373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59d4104240 0x7f59d419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.322+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 >> 172.21.15.16:0/840624373 conn(0x7f59d40ff7b0 msgr2=0x7f59d41086d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.323+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 shutdown_connections 2024-04-13T03:31:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.323+0000 7f59d9000640 1 -- 172.21.15.16:0/840624373 wait complete. 2024-04-13T03:31:43.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:31:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.541+0000 7fe806a00640 1 -- 172.21.15.16:0/3238711563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe800103bc0 msgr2=0x7fe80010a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.541+0000 7fe806a00640 1 --2- 172.21.15.16:0/3238711563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe800103bc0 0x7fe80010a450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009be0 tx=0x7fe7f802bef0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.541+0000 7fe806a00640 1 -- 172.21.15.16:0/3238711563 shutdown_connections 2024-04-13T03:31:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.541+0000 7fe806a00640 1 --2- 172.21.15.16:0/3238711563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe800103bc0 0x7fe80010a450 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.541+0000 7fe806a00640 1 --2- 172.21.15.16:0/3238711563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe800103200 0x7fe800103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.541+0000 7fe806a00640 1 --2- 172.21.15.16:0/3238711563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe800102010 0x7fe800102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.541+0000 7fe806a00640 1 -- 172.21.15.16:0/3238711563 >> 172.21.15.16:0/3238711563 conn(0x7fe8000fd5a0 msgr2=0x7fe8000ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.542+0000 7fe806a00640 1 -- 172.21.15.16:0/3238711563 shutdown_connections 2024-04-13T03:31:43.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.542+0000 7fe806a00640 1 -- 172.21.15.16:0/3238711563 wait complete. 2024-04-13T03:31:43.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.542+0000 7fe806a00640 1 Processor -- start 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe806a00640 1 -- start start 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe806a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe800102010 0x7fe8001982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe806a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe800103200 0x7fe800198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe806a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe800103bc0 0x7fe80019f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe806a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe80010dcf0 con 0x7fe800103200 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe806a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe80010db70 con 0x7fe800103bc0 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe806a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe80010de70 con 0x7fe800102010 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe7ff400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe800103200 0x7fe800198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe7ff400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe800103200 0x7fe800198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53548/0 (socket says 172.21.15.16:53548) 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe7ff400640 1 -- 172.21.15.16:0/3797089103 learned_addr learned my addr 172.21.15.16:0/3797089103 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe7ffe00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe800102010 0x7fe8001982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:43.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.543+0000 7fe804a00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe800103bc0 0x7fe80019f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe7ff400640 1 -- 172.21.15.16:0/3797089103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe800102010 msgr2=0x7fe8001982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe7ff400640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe800102010 0x7fe8001982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe7ff400640 1 -- 172.21.15.16:0/3797089103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe800103bc0 msgr2=0x7fe80019f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe7ff400640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe800103bc0 0x7fe80019f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe7ff400640 1 -- 172.21.15.16:0/3797089103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe80019ff10 con 0x7fe800103200 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe7ff400640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe800103200 0x7fe800198830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800e9f0 tx=0x7fe7e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe7fcc00640 1 -- 172.21.15.16:0/3797089103 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800cd80 con 0x7fe800103200 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe7fcc00640 1 -- 172.21.15.16:0/3797089103 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e8010910 con 0x7fe800103200 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8001a0200 con 0x7fe800103200 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe7fcc00640 1 -- 172.21.15.16:0/3797089103 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8018990 con 0x7fe800103200 2024-04-13T03:31:43.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.544+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8001a07a0 con 0x7fe800103200 2024-04-13T03:31:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.546+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe800109400 con 0x7fe800103200 2024-04-13T03:31:43.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.548+0000 7fe7fcc00640 1 -- 172.21.15.16:0/3797089103 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe7e8010ac0 con 0x7fe800103200 2024-04-13T03:31:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.549+0000 7fe7fcc00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe7e40a3c30 0x7fe7e40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:43.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.549+0000 7fe7fcc00640 1 -- 172.21.15.16:0/3797089103 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe7e80d0910 con 0x7fe800103200 2024-04-13T03:31:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.549+0000 7fe7ffe00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe7e40a3c30 0x7fe7e40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:43.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.550+0000 7fe7ffe00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe7e40a3c30 0x7fe7e40a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009530 tx=0x7fe7f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:43.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.553+0000 7fe7fcc00640 1 -- 172.21.15.16:0/3797089103 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7e809ca00 con 0x7fe800103200 2024-04-13T03:31:43.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.754+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe800062bf0 con 0x7fe800103200 2024-04-13T03:31:43.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.756+0000 7fe7fcc00640 1 -- 172.21.15.16:0/3797089103 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7fe7e80a0d00 con 0x7fe800103200 2024-04-13T03:31:43.757 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:43.758 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:31:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe7e40a3c30 msgr2=0x7fe7e40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe7e40a3c30 0x7fe7e40a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009530 tx=0x7fe7f0009260 comp rx=0 tx=0).stop 2024-04-13T03:31:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe800103200 msgr2=0x7fe800198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe800103200 0x7fe800198830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800e9f0 tx=0x7fe7e800eeb0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 shutdown_connections 2024-04-13T03:31:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe7e40a3c30 0x7fe7e40a60f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe800103bc0 0x7fe80019f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe800103200 0x7fe800198830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 --2- 172.21.15.16:0/3797089103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe800102010 0x7fe8001982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 >> 172.21.15.16:0/3797089103 conn(0x7fe8000fd5a0 msgr2=0x7fe8001065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 shutdown_connections 2024-04-13T03:31:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:43.759+0000 7fe806a00640 1 -- 172.21.15.16:0/3797089103 wait complete. 2024-04-13T03:31:43.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:31:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.002+0000 7fd09f200640 1 -- 172.21.15.16:0/3177933695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0109630 msgr2=0x7fd0a0109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.002+0000 7fd09f200640 1 --2- 172.21.15.16:0/3177933695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0109630 0x7fd0a0109ab0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009900 tx=0x7fd08c02bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.002+0000 7fd09f200640 1 -- 172.21.15.16:0/3177933695 shutdown_connections 2024-04-13T03:31:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.002+0000 7fd09f200640 1 --2- 172.21.15.16:0/3177933695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0a0109ff0 0x7fd0a0110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.002+0000 7fd09f200640 1 --2- 172.21.15.16:0/3177933695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0109630 0x7fd0a0109ab0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.002+0000 7fd09f200640 1 --2- 172.21.15.16:0/3177933695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a0108440 0x7fd0a0108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.002+0000 7fd09f200640 1 -- 172.21.15.16:0/3177933695 >> 172.21.15.16:0/3177933695 conn(0x7fd0a0077400 msgr2=0x7fd0a0079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.002+0000 7fd09f200640 1 -- 172.21.15.16:0/3177933695 shutdown_connections 2024-04-13T03:31:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.002+0000 7fd09f200640 1 -- 172.21.15.16:0/3177933695 wait complete. 2024-04-13T03:31:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.003+0000 7fd09f200640 1 Processor -- start 2024-04-13T03:31:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.003+0000 7fd09f200640 1 -- start start 2024-04-13T03:31:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.003+0000 7fd09f200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0108440 0x7fd0a019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.003+0000 7fd09f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a0109630 0x7fd0a019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.003+0000 7fd09f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0a0109ff0 0x7fd0a01a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01140d0 con 0x7fd0a0109630 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09f200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a0113f50 con 0x7fd0a0108440 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a0114250 con 0x7fd0a0109ff0 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0108440 0x7fd0a019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0108440 0x7fd0a019e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:54084/0 (socket says 172.21.15.16:54084) 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09de00640 1 -- 172.21.15.16:0/2128790924 learned_addr learned my addr 172.21.15.16:0/2128790924 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09d400640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a0109630 0x7fd0a019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09e800640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0a0109ff0 0x7fd0a01a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09de00640 1 -- 172.21.15.16:0/2128790924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0a0109ff0 msgr2=0x7fd0a01a5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09de00640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0a0109ff0 0x7fd0a01a5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09de00640 1 -- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a0109630 msgr2=0x7fd0a019ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09de00640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a0109630 0x7fd0a019ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09de00640 1 -- 172.21.15.16:0/2128790924 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a01a6350 con 0x7fd0a0108440 2024-04-13T03:31:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09e800640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0a0109ff0 0x7fd0a01a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.004+0000 7fd09d400640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a0109630 0x7fd0a019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.005+0000 7fd09de00640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0108440 0x7fd0a019e780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd09000e9c0 tx=0x7fd09000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.005+0000 7fd096a00640 1 -- 172.21.15.16:0/2128790924 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09000cd80 con 0x7fd0a0108440 2024-04-13T03:31:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.005+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a01a6640 con 0x7fd0a0108440 2024-04-13T03:31:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.005+0000 7fd096a00640 1 -- 172.21.15.16:0/2128790924 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0900035f0 con 0x7fd0a0108440 2024-04-13T03:31:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.005+0000 7fd096a00640 1 -- 172.21.15.16:0/2128790924 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd090010690 con 0x7fd0a0108440 2024-04-13T03:31:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.005+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a01a6b60 con 0x7fd0a0108440 2024-04-13T03:31:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.006+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a010f760 con 0x7fd0a0108440 2024-04-13T03:31:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.009+0000 7fd096a00640 1 -- 172.21.15.16:0/2128790924 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd0900037a0 con 0x7fd0a0108440 2024-04-13T03:31:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.010+0000 7fd096a00640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd0840a3bb0 0x7fd0840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.010+0000 7fd096a00640 1 -- 172.21.15.16:0/2128790924 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd0900d0b10 con 0x7fd0a0108440 2024-04-13T03:31:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.010+0000 7fd09d400640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd0840a3bb0 0x7fd0840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.011+0000 7fd09d400640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd0840a3bb0 0x7fd0840a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd08c02c7b0 tx=0x7fd08c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:44.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.016+0000 7fd096a00640 1 -- 172.21.15.16:0/2128790924 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd09009d2e0 con 0x7fd0a0108440 2024-04-13T03:31:44.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:43 smithi016 ceph-mon[31772]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-13T03:31:44.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:43 smithi016 ceph-mon[31772]: osdmap e90: 12 total, 12 up, 12 in 2024-04-13T03:31:44.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:43 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2854805657' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-13T03:31:44.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:43 smithi016 ceph-mon[31772]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:31:44.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:43 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/840624373' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:44.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:43 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3797089103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:44.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:43 smithi188 ceph-mon[36398]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-13T03:31:44.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:43 smithi188 ceph-mon[36398]: osdmap e90: 12 total, 12 up, 12 in 2024-04-13T03:31:44.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:43 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2854805657' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-13T03:31:44.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:43 smithi188 ceph-mon[36398]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:31:44.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:43 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/840624373' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:44.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:43 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3797089103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:44.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.224+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0a0062bc0 con 0x7fd0a0108440 2024-04-13T03:31:44.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.227+0000 7fd096a00640 1 -- 172.21.15.16:0/2128790924 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7fd09009cbe0 con 0x7fd0a0108440 2024-04-13T03:31:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:44.228 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:31:44.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.229+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd0840a3bb0 msgr2=0x7fd0840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.229+0000 7fd09f200640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd0840a3bb0 0x7fd0840a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd08c02c7b0 tx=0x7fd08c005fd0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.229+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0108440 msgr2=0x7fd0a019e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.229+0000 7fd09f200640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0108440 0x7fd0a019e780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd09000e9c0 tx=0x7fd09000ee80 comp rx=0 tx=0).stop 2024-04-13T03:31:44.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.230+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 shutdown_connections 2024-04-13T03:31:44.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.230+0000 7fd09f200640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd0840a3bb0 0x7fd0840a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.230+0000 7fd09f200640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0a0109ff0 0x7fd0a01a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.230+0000 7fd09f200640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0a0109630 0x7fd0a019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.230+0000 7fd09f200640 1 --2- 172.21.15.16:0/2128790924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0a0108440 0x7fd0a019e780 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.230+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 >> 172.21.15.16:0/2128790924 conn(0x7fd0a0077400 msgr2=0x7fd0a010c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.230+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 shutdown_connections 2024-04-13T03:31:44.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.230+0000 7fd09f200640 1 -- 172.21.15.16:0/2128790924 wait complete. 2024-04-13T03:31:44.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:31:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:43 smithi119 ceph-mon[35835]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-13T03:31:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:43 smithi119 ceph-mon[35835]: osdmap e90: 12 total, 12 up, 12 in 2024-04-13T03:31:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:43 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2854805657' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-13T03:31:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:43 smithi119 ceph-mon[35835]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:31:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:43 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/840624373' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:44.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:43 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3797089103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.483+0000 7faeb3e00640 1 -- 172.21.15.16:0/1978434227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 msgr2=0x7faeb4104460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.483+0000 7faeb3e00640 1 --2- 172.21.15.16:0/1978434227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 0x7faeb4104460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faea4009860 tx=0x7faea402bb70 comp rx=0 tx=0).stop 2024-04-13T03:31:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.484+0000 7faeb3e00640 1 -- 172.21.15.16:0/1978434227 shutdown_connections 2024-04-13T03:31:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.484+0000 7faeb3e00640 1 --2- 172.21.15.16:0/1978434227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faeb4105bf0 0x7faeb410c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.484+0000 7faeb3e00640 1 --2- 172.21.15.16:0/1978434227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faeb4105230 0x7faeb41056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.484+0000 7faeb3e00640 1 --2- 172.21.15.16:0/1978434227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 0x7faeb4104460 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.484+0000 7faeb3e00640 1 -- 172.21.15.16:0/1978434227 >> 172.21.15.16:0/1978434227 conn(0x7faeb40ff5d0 msgr2=0x7faeb41019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.484+0000 7faeb3e00640 1 -- 172.21.15.16:0/1978434227 shutdown_connections 2024-04-13T03:31:44.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.484+0000 7faeb3e00640 1 -- 172.21.15.16:0/1978434227 wait complete. 2024-04-13T03:31:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.485+0000 7faeb3e00640 1 Processor -- start 2024-04-13T03:31:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.485+0000 7faeb3e00640 1 -- start start 2024-04-13T03:31:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.485+0000 7faeb3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 0x7faeb419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.485+0000 7faeb3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faeb4105230 0x7faeb419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.485+0000 7faeb3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faeb4105bf0 0x7faeb41a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.485+0000 7faeb3e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faeb410fcd0 con 0x7faeb4105bf0 2024-04-13T03:31:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.485+0000 7faeb3e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7faeb410fb50 con 0x7faeb4104040 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.485+0000 7faeb3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faeb410fe50 con 0x7faeb4105230 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 0x7faeb419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faeb4105bf0 0x7faeb41a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 0x7faeb419e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:54112/0 (socket says 172.21.15.16:54112) 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb2a00640 1 -- 172.21.15.16:0/573808799 learned_addr learned my addr 172.21.15.16:0/573808799 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeabe00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faeb4105230 0x7faeb419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb2a00640 1 -- 172.21.15.16:0/573808799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faeb4105230 msgr2=0x7faeb419ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb2a00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faeb4105230 0x7faeb419ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb2a00640 1 -- 172.21.15.16:0/573808799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faeb4105bf0 msgr2=0x7faeb41a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb2a00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faeb4105bf0 0x7faeb41a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb2a00640 1 -- 172.21.15.16:0/573808799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb41a6340 con 0x7faeb4104040 2024-04-13T03:31:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.486+0000 7faeb2a00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 0x7faeb419e720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faea40349e0 tx=0x7faea4032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.487+0000 7faeab400640 1 -- 172.21.15.16:0/573808799 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea4041020 con 0x7faeb4104040 2024-04-13T03:31:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.487+0000 7faeab400640 1 -- 172.21.15.16:0/573808799 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faea402c9f0 con 0x7faeb4104040 2024-04-13T03:31:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.487+0000 7faeab400640 1 -- 172.21.15.16:0/573808799 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea4039040 con 0x7faeb4104040 2024-04-13T03:31:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.487+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb41a65d0 con 0x7faeb4104040 2024-04-13T03:31:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.487+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb41a6a70 con 0x7faeb4104040 2024-04-13T03:31:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.488+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb41a6cd0 con 0x7faeb4104040 2024-04-13T03:31:44.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.490+0000 7faeab400640 1 -- 172.21.15.16:0/573808799 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faea4044020 con 0x7faeb4104040 2024-04-13T03:31:44.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.491+0000 7faeab400640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fae980a3c30 0x7fae980a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.491+0000 7faeab400640 1 -- 172.21.15.16:0/573808799 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7faea40f2200 con 0x7faeb4104040 2024-04-13T03:31:44.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.492+0000 7faeabe00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fae980a3c30 0x7fae980a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.492+0000 7faeabe00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fae980a3c30 0x7fae980a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faeb419fc30 tx=0x7fae9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:44.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.493+0000 7faeab400640 1 -- 172.21.15.16:0/573808799 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faea404b020 con 0x7faeb4104040 2024-04-13T03:31:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.738+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faeb41a6ea0 con 0x7faeb4104040 2024-04-13T03:31:44.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.739+0000 7faeab400640 1 -- 172.21.15.16:0/573808799 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11797 (secure 0 0 0) 0x7faea40be9c0 con 0x7faeb4104040 2024-04-13T03:31:44.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:44.741 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24472,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24523,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:31:42.303966+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24587},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24587,"qdb_cluster":[24587]},"id":1}]} 2024-04-13T03:31:44.741 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2024-04-13T03:31:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fae980a3c30 msgr2=0x7fae980a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fae980a3c30 0x7fae980a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faeb419fc30 tx=0x7fae9c005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 msgr2=0x7faeb419e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 0x7faeb419e720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faea40349e0 tx=0x7faea4032430 comp rx=0 tx=0).stop 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 shutdown_connections 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fae980a3c30 0x7fae980a60f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faeb4105bf0 0x7faeb41a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faeb4105230 0x7faeb419ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 --2- 172.21.15.16:0/573808799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faeb4104040 0x7faeb419e720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 >> 172.21.15.16:0/573808799 conn(0x7faeb40ff5d0 msgr2=0x7faeb4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 shutdown_connections 2024-04-13T03:31:44.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.742+0000 7faeb3e00640 1 -- 172.21.15.16:0/573808799 wait complete. 2024-04-13T03:31:44.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:31:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.973+0000 7f3f87200640 1 -- 172.21.15.16:0/692360153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f80105430 msgr2=0x7f3f801058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.973+0000 7f3f87200640 1 --2- 172.21.15.16:0/692360153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f80105430 0x7f3f801058b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009ca0 tx=0x7f3f7002c100 comp rx=0 tx=0).stop 2024-04-13T03:31:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.973+0000 7f3f87200640 1 -- 172.21.15.16:0/692360153 shutdown_connections 2024-04-13T03:31:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.973+0000 7f3f87200640 1 --2- 172.21.15.16:0/692360153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80105df0 0x7f3f8010c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.973+0000 7f3f87200640 1 --2- 172.21.15.16:0/692360153 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f80105430 0x7f3f801058b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.973+0000 7f3f87200640 1 --2- 172.21.15.16:0/692360153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f80104240 0x7f3f80104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.973+0000 7f3f87200640 1 -- 172.21.15.16:0/692360153 >> 172.21.15.16:0/692360153 conn(0x7f3f800ff7b0 msgr2=0x7f3f80101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.973+0000 7f3f87200640 1 -- 172.21.15.16:0/692360153 shutdown_connections 2024-04-13T03:31:44.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.973+0000 7f3f87200640 1 -- 172.21.15.16:0/692360153 wait complete. 2024-04-13T03:31:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.974+0000 7f3f87200640 1 Processor -- start 2024-04-13T03:31:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.974+0000 7f3f87200640 1 -- start start 2024-04-13T03:31:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f87200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f80104240 0x7f3f8019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f87200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80105430 0x7f3f8019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f87200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f80105df0 0x7f3f801a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f87200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8010fed0 con 0x7f3f80105df0 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f87200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8010fd50 con 0x7f3f80105430 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f87200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3f80110050 con 0x7f3f80104240 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f7f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80105430 0x7f3f8019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f85200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f80105df0 0x7f3f801a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f7f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80105430 0x7f3f8019aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:54130/0 (socket says 172.21.15.16:54130) 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f7f400640 1 -- 172.21.15.16:0/4137157914 learned_addr learned my addr 172.21.15.16:0/4137157914 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f85200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f80105df0 0x7f3f801a1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53592/0 (socket says 172.21.15.16:53592) 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f7fe00640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f80104240 0x7f3f8019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f7f400640 1 -- 172.21.15.16:0/4137157914 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f80104240 msgr2=0x7f3f8019a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f7f400640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f80104240 0x7f3f8019a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f7f400640 1 -- 172.21.15.16:0/4137157914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f80105df0 msgr2=0x7f3f801a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:44.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.975+0000 7f3f7f400640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f80105df0 0x7f3f801a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.976+0000 7f3f7f400640 1 -- 172.21.15.16:0/4137157914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f801a2130 con 0x7f3f80105430 2024-04-13T03:31:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.976+0000 7f3f7f400640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80105430 0x7f3f8019aa50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3f700325f0 tx=0x7f3f70035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.976+0000 7f3f7cc00640 1 -- 172.21.15.16:0/4137157914 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70039040 con 0x7f3f80105430 2024-04-13T03:31:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.976+0000 7f3f7cc00640 1 -- 172.21.15.16:0/4137157914 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f70044070 con 0x7f3f80105430 2024-04-13T03:31:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.976+0000 7f3f7cc00640 1 -- 172.21.15.16:0/4137157914 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f700044d0 con 0x7f3f80105430 2024-04-13T03:31:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.976+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f801a23c0 con 0x7f3f80105430 2024-04-13T03:31:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.976+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f801a2830 con 0x7f3f80105430 2024-04-13T03:31:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.978+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f8010b630 con 0x7f3f80105430 2024-04-13T03:31:44.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.980+0000 7f3f7cc00640 1 -- 172.21.15.16:0/4137157914 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3f70004670 con 0x7f3f80105430 2024-04-13T03:31:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.981+0000 7f3f7cc00640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f640a3bb0 0x7f3f640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.981+0000 7f3f7cc00640 1 -- 172.21.15.16:0/4137157914 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3f700f2be0 con 0x7f3f80105430 2024-04-13T03:31:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.981+0000 7f3f7fe00640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f640a3bb0 0x7f3f640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.982+0000 7f3f7fe00640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f640a3bb0 0x7f3f640a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3f80105290 tx=0x7f3f68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:44.985+0000 7f3f7cc00640 1 -- 172.21.15.16:0/4137157914 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f700bf380 con 0x7f3f80105430 2024-04-13T03:31:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:44 smithi016 ceph-mon[31772]: pgmap v322: 129 pgs: 104 active+clean, 16 creating+peering, 9 unknown; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-04-13T03:31:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:44 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2128790924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:44 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/573808799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:45.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:44 smithi188 ceph-mon[36398]: pgmap v322: 129 pgs: 104 active+clean, 16 creating+peering, 9 unknown; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-04-13T03:31:45.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:44 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2128790924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:45.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:44 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/573808799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:45.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.195+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f80062bf0 con 0x7f3f80105430 2024-04-13T03:31:45.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.198+0000 7f3f7cc00640 1 -- 172.21.15.16:0/4137157914 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7f3f700bec80 con 0x7f3f80105430 2024-04-13T03:31:45.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:45.199 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:31:45.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f640a3bb0 msgr2=0x7f3f640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f640a3bb0 0x7f3f640a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3f80105290 tx=0x7f3f68005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:45.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80105430 msgr2=0x7f3f8019aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80105430 0x7f3f8019aa50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3f700325f0 tx=0x7f3f70035d10 comp rx=0 tx=0).stop 2024-04-13T03:31:45.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 shutdown_connections 2024-04-13T03:31:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f640a3bb0 0x7f3f640a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f80105df0 0x7f3f801a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f80105430 0x7f3f8019aa50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 --2- 172.21.15.16:0/4137157914 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f80104240 0x7f3f8019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 >> 172.21.15.16:0/4137157914 conn(0x7f3f800ff7b0 msgr2=0x7f3f80108800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 shutdown_connections 2024-04-13T03:31:45.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.201+0000 7f3f87200640 1 -- 172.21.15.16:0/4137157914 wait complete. 2024-04-13T03:31:45.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-13T03:31:37.109492+0000', 'modified': '2024-04-13T03:31:42.303966+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24587}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/3722020849', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 3722020849}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 3722020849}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24587, 'qdb_cluster': [24587]} 2024-04-13T03:31:45.222 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-13T03:31:45.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-13T03:31:45.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:44 smithi119 ceph-mon[35835]: pgmap v322: 129 pgs: 104 active+clean, 16 creating+peering, 9 unknown; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-04-13T03:31:45.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:44 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2128790924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:45.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:44 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/573808799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:31:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.442+0000 7fa66fe00640 1 -- 172.21.15.16:0/908526649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6701096f0 msgr2=0x7fa670109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.442+0000 7fa66fe00640 1 --2- 172.21.15.16:0/908526649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6701096f0 0x7fa670109b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009900 tx=0x7fa65c02bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.442+0000 7fa66fe00640 1 -- 172.21.15.16:0/908526649 shutdown_connections 2024-04-13T03:31:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.442+0000 7fa66fe00640 1 --2- 172.21.15.16:0/908526649 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa67010a0b0 0x7fa670110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.442+0000 7fa66fe00640 1 --2- 172.21.15.16:0/908526649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6701096f0 0x7fa670109b70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.442+0000 7fa66fe00640 1 --2- 172.21.15.16:0/908526649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa670108500 0x7fa670108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.442+0000 7fa66fe00640 1 -- 172.21.15.16:0/908526649 >> 172.21.15.16:0/908526649 conn(0x7fa670077490 msgr2=0x7fa6700798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.442+0000 7fa66fe00640 1 -- 172.21.15.16:0/908526649 shutdown_connections 2024-04-13T03:31:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.442+0000 7fa66fe00640 1 -- 172.21.15.16:0/908526649 wait complete. 2024-04-13T03:31:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.443+0000 7fa66fe00640 1 Processor -- start 2024-04-13T03:31:45.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.443+0000 7fa66fe00640 1 -- start start 2024-04-13T03:31:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa670108500 0x7fa67019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6701096f0 0x7fa67019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa67010a0b0 0x7fa6701a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66fe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa670114190 con 0x7fa6701096f0 2024-04-13T03:31:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa670114010 con 0x7fa670108500 2024-04-13T03:31:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa670114310 con 0x7fa67010a0b0 2024-04-13T03:31:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6701096f0 0x7fa67019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa670108500 0x7fa67019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa670108500 0x7fa67019e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:54152/0 (socket says 172.21.15.16:54152) 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66ea00640 1 -- 172.21.15.16:0/91538473 learned_addr learned my addr 172.21.15.16:0/91538473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.444+0000 7fa66f400640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa67010a0b0 0x7fa6701a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.445+0000 7fa66e000640 1 -- 172.21.15.16:0/91538473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa67010a0b0 msgr2=0x7fa6701a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.445+0000 7fa66e000640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa67010a0b0 0x7fa6701a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.445+0000 7fa66e000640 1 -- 172.21.15.16:0/91538473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa670108500 msgr2=0x7fa67019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.445+0000 7fa66e000640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa670108500 0x7fa67019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.445+0000 7fa66e000640 1 -- 172.21.15.16:0/91538473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6701a63a0 con 0x7fa6701096f0 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.445+0000 7fa66f400640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa67010a0b0 0x7fa6701a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.445+0000 7fa66ea00640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa670108500 0x7fa67019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:45.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.445+0000 7fa66e000640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6701096f0 0x7fa67019ed60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009230 tx=0x7fa65c02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.445+0000 7fa667400640 1 -- 172.21.15.16:0/91538473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c041070 con 0x7fa6701096f0 2024-04-13T03:31:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.446+0000 7fa667400640 1 -- 172.21.15.16:0/91538473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa65c004450 con 0x7fa6701096f0 2024-04-13T03:31:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.446+0000 7fa66fe00640 1 -- 172.21.15.16:0/91538473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6701a6630 con 0x7fa6701096f0 2024-04-13T03:31:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.446+0000 7fa667400640 1 -- 172.21.15.16:0/91538473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c034440 con 0x7fa6701096f0 2024-04-13T03:31:45.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.446+0000 7fa66fe00640 1 -- 172.21.15.16:0/91538473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6701a6b20 con 0x7fa6701096f0 2024-04-13T03:31:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.449+0000 7fa667400640 1 -- 172.21.15.16:0/91538473 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa65c044070 con 0x7fa6701096f0 2024-04-13T03:31:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.449+0000 7fa6785f6740 1 Processor -- start 2024-04-13T03:31:45.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.449+0000 7fa6785f6740 1 -- start start 2024-04-13T03:31:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c75bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c761a0 0x55c796c7aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c7b310 0x55c796c7d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa6785f6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c796bdb990 con 0x55c796c7b310 2024-04-13T03:31:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa6785f6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c796b82530 con 0x55c796c757b0 2024-04-13T03:31:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa6785f6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c796bd7950 con 0x55c796c761a0 2024-04-13T03:31:45.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa66f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c75bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa66f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c75bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:54162/0 (socket says 172.21.15.16:54162) 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa66f400640 1 -- 172.21.15.16:0/4024212403 learned_addr learned my addr 172.21.15.16:0/4024212403 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa66e000640 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c7b310 0x55c796c7d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa66ea00640 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c761a0 0x55c796c7aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.450+0000 7fa66f400640 1 -- 172.21.15.16:0/4024212403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c761a0 msgr2=0x55c796c7aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa66f400640 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c761a0 0x55c796c7aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa66f400640 1 -- 172.21.15.16:0/4024212403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c7b310 msgr2=0x55c796c7d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa66f400640 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c7b310 0x55c796c7d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa66f400640 1 -- 172.21.15.16:0/4024212403 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7968ee2d0 con 0x55c796c757b0 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa66ea00640 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c761a0 0x55c796c7aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa66e000640 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c7b310 0x55c796c7d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa66f400640 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c75bd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6680117e0 tx=0x7fa668011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a486e82617cd42f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa65be00640 1 -- 172.21.15.16:0/4024212403 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668011310 con 0x55c796c757b0 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa65be00640 1 -- 172.21.15.16:0/4024212403 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa668003e50 con 0x55c796c757b0 2024-04-13T03:31:45.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa65be00640 1 -- 172.21.15.16:0/4024212403 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668004590 con 0x55c796c757b0 2024-04-13T03:31:45.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.451+0000 7fa667400640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa6540a3bb0 0x7fa6540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa667400640 1 -- 172.21.15.16:0/91538473 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa65c0f2140 con 0x7fa6701096f0 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa66ea00640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa6540a3bb0 0x7fa6540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa6785f6740 1 -- 172.21.15.16:0/4024212403 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 msgr2=0x55c796c75bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa6785f6740 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c75bd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6680117e0 tx=0x7fa668011ca0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa6785f6740 1 -- 172.21.15.16:0/4024212403 shutdown_connections 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa6785f6740 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c7b310 0x55c796c7d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa6785f6740 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c761a0 0x55c796c7aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa6785f6740 1 --2- 172.21.15.16:0/4024212403 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c75bd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa6785f6740 1 -- 172.21.15.16:0/4024212403 >> 172.21.15.16:0/4024212403 conn(0x55c796c70d20 msgr2=0x55c796c71120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa6785f6740 1 -- 172.21.15.16:0/4024212403 shutdown_connections 2024-04-13T03:31:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa6785f6740 1 -- 172.21.15.16:0/4024212403 wait complete. 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.452+0000 7fa66ea00640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa6540a3bb0 0x7fa6540a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55c796c765a0 tx=0x7fa66000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.453+0000 7fa6785f6740 1 -- start start 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.453+0000 7fa6785f6740 1 Processor -- start 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.453+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c757b0 0x55c796c7fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.453+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c80270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.453+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c7b310 0x55c796c872b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.453+0000 7fa6785f6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c796c7e2c0 con 0x55c796c761a0 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.453+0000 7fa6785f6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c796b958f0 con 0x55c796c7b310 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.453+0000 7fa6785f6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c796c7e440 con 0x55c796c757b0 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa6785f6740 10 client.0 _ll_register_callbacks cb 0x55c796b952e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c80270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c80270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53640/0 (socket says 172.21.15.16:53640) 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66e000640 1 -- 172.21.15.16:0/2960144358 learned_addr learned my addr 172.21.15.16:0/2960144358 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66f400640 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c7b310 0x55c796c872b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66ea00640 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c757b0 0x55c796c7fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66e000640 1 -- 172.21.15.16:0/2960144358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c757b0 msgr2=0x55c796c7fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66e000640 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c757b0 0x55c796c7fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66e000640 1 -- 172.21.15.16:0/2960144358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c7b310 msgr2=0x55c796c872b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66e000640 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c7b310 0x55c796c872b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66e000640 1 -- 172.21.15.16:0/2960144358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7969b86b0 con 0x55c796c761a0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa66e000640 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c80270 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0ba6f0 tx=0x7fa65c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abb3ea5eb30613d server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.454+0000 7fa658c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.455+0000 7fa658c00640 1 -- 172.21.15.16:0/2960144358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c04b040 con 0x55c796c761a0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.455+0000 7fa658c00640 1 -- 172.21.15.16:0/2960144358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa65c0035f0 con 0x55c796c761a0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.455+0000 7fa6785f6740 1 -- 172.21.15.16:0/2960144358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa668003e50 con 0x55c796c761a0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.455+0000 7fa658c00640 1 -- 172.21.15.16:0/2960144358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fa65c0bddd0 con 0x55c796c761a0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.455+0000 7fa6785f6740 10 client.14589 fetch_fsmap learned FSMap version 16 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.455+0000 7fa6785f6740 1 -- 172.21.15.16:0/2960144358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55c796c8a620 con 0x55c796c761a0 2024-04-13T03:31:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.455+0000 7fa658c00640 1 -- 172.21.15.16:0/2960144358 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa668003e50 con 0x55c796c761a0 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.455+0000 7fa658c00640 1 -- 172.21.15.16:0/2960144358 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fa65c0c2120 con 0x55c796c761a0 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.456+0000 7fa6785f6740 10 client.14589 fetch_fsmap finished waiting for FSMap version 16 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.456+0000 7fa6785f6740 10 client.14589 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.456+0000 7fa6785f6740 20 client.14589 populate_metadata read hostname 'smithi016' 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.456+0000 7fa6785f6740 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55c796c8cca0 0x55c796c8e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.456+0000 7fa6785f6740 4 client.14589 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.456+0000 7fa6785f6740 1 -- 172.21.15.16:0/2960144358 --> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c796b81b40 con 0x55c796c8cca0 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.456+0000 7fa66ea00640 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55c796c8cca0 0x55c796c8e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.457+0000 7fa66ea00640 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55c796c8cca0 0x55c796c8e110 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87ca4144bfdf4221 server_cookie=4e9e50ed3d2e02bf in_seq=0 out_seq=0 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.457+0000 7fa658c00640 10 client.14589 ms_handle_connect on v2:172.21.15.119:6838/3722020849 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa658c00640 1 -- 172.21.15.16:0/2960144358 <== mds.0 v2:172.21.15.119:6838/3722020849 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c796b81b40 con 0x55c796c8cca0 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa658c00640 10 client.14589 handle_command_reply: tid=0 2024-04-13T03:31:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 client.14589 shutdown 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 -- 172.21.15.16:0/2960144358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 msgr2=0x55c796c80270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c80270 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0ba6f0 tx=0x7fa65c000bc0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 -- 172.21.15.16:0/2960144358 shutdown_connections 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55c796c8cca0 0x55c796c8e110 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c7b310 0x55c796c872b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c80270 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa658c00640 0 client.14589 ms_handle_reset on v2:172.21.15.119:6838/3722020849 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 --2- 172.21.15.16:0/2960144358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c757b0 0x55c796c7fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 -- 172.21.15.16:0/2960144358 >> 172.21.15.16:0/2960144358 conn(0x55c796c70d20 msgr2=0x55c796c6ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.458+0000 7fa6785f6740 1 -- 172.21.15.16:0/2960144358 shutdown_connections 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.459+0000 7fa6785f6740 1 -- 172.21.15.16:0/2960144358 wait complete. 2024-04-13T03:31:45.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.459+0000 7fa6785f6740 20 client.14589 trim_cache size 0 max 16384 2024-04-13T03:31:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.479+0000 7fa6785f6740 1 Processor -- start 2024-04-13T03:31:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.479+0000 7fa6785f6740 1 -- start start 2024-04-13T03:31:45.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.479+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c7e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.479+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c87270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.479+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 0x55c796c8b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.479+0000 7fa6785f6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c796c8c450 con 0x55c796c761a0 2024-04-13T03:31:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.479+0000 7fa6785f6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c796c8cb20 con 0x55c796c757b0 2024-04-13T03:31:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.479+0000 7fa6785f6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c796c8d1f0 con 0x55c796c7b310 2024-04-13T03:31:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.479+0000 7fa66e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c87270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c87270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53646/0 (socket says 172.21.15.16:53646) 2024-04-13T03:31:45.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66e000640 1 -- 172.21.15.16:0/3442334248 learned_addr learned my addr 172.21.15.16:0/3442334248 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66ea00640 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c7e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66f400640 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 0x55c796c8b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66e000640 1 -- 172.21.15.16:0/3442334248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 msgr2=0x55c796c8b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:31:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66e000640 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 0x55c796c8b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66e000640 1 -- 172.21.15.16:0/3442334248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 msgr2=0x55c796c7e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:31:45.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66e000640 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c7e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66e000640 1 -- 172.21.15.16:0/3442334248 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c796c7a980 con 0x55c796c761a0 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66ea00640 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c7e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66f400640 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 0x55c796c8b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa66e000640 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c87270 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0bac60 tx=0x7fa65c0bc7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df0f973b75040f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.480+0000 7fa65b400640 1 -- 172.21.15.16:0/3442334248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0d7070 con 0x55c796c761a0 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa65b400640 1 -- 172.21.15.16:0/3442334248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa65c0be290 con 0x55c796c761a0 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa65b400640 1 -- 172.21.15.16:0/3442334248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c04b070 con 0x55c796c761a0 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa6785f6740 1 -- 172.21.15.16:0/3442334248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 msgr2=0x55c796c87270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa6785f6740 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c87270 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0bac60 tx=0x7fa65c0bc7a0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa6785f6740 1 -- 172.21.15.16:0/3442334248 shutdown_connections 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa6785f6740 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 0x55c796c8b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa6785f6740 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c761a0 0x55c796c87270 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa6785f6740 1 --2- 172.21.15.16:0/3442334248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c757b0 0x55c796c7e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa6785f6740 1 -- 172.21.15.16:0/3442334248 >> 172.21.15.16:0/3442334248 conn(0x55c796c70d20 msgr2=0x55c796c71100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:45.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa6785f6740 1 -- 172.21.15.16:0/3442334248 shutdown_connections 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.481+0000 7fa6785f6740 1 -- 172.21.15.16:0/3442334248 wait complete. 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.482+0000 7fa6785f6740 1 -- start start 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.482+0000 7fa6785f6740 1 Processor -- start 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.482+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c757b0 0x55c796c79870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.482+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c761a0 0x55c796c79db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.482+0000 7fa6785f6740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 0x55c796c85950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.482+0000 7fa6785f6740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55c796c8df90 con 0x55c796c757b0 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.482+0000 7fa6785f6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c796c8d8c0 con 0x55c796c761a0 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.482+0000 7fa6785f6740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55c796c8e660 con 0x55c796c7b310 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.482+0000 7fa6785f6740 10 client.0 _ll_register_callbacks cb 0x7fa65c003f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c757b0 0x55c796c79870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c757b0 0x55c796c79870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53658/0 (socket says 172.21.15.16:53658) 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66ea00640 1 -- 172.21.15.16:0/636130723 learned_addr learned my addr 172.21.15.16:0/636130723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:45.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66e000640 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c761a0 0x55c796c79db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66f400640 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 0x55c796c85950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66ea00640 1 -- 172.21.15.16:0/636130723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 msgr2=0x55c796c85950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:31:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66ea00640 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 0x55c796c85950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66ea00640 1 -- 172.21.15.16:0/636130723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c761a0 msgr2=0x55c796c79db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:31:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66ea00640 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c761a0 0x55c796c79db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.483+0000 7fa66ea00640 1 -- 172.21.15.16:0/636130723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c796c85f40 con 0x55c796c757b0 2024-04-13T03:31:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.484+0000 7fa66ea00640 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c757b0 0x55c796c79870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa66000b2b0 tx=0x7fa660014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d705d41523397a1e server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.484+0000 7fa658c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:31:45.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.484+0000 7fa658c00640 1 -- 172.21.15.16:0/636130723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66001ea40 con 0x55c796c757b0 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.484+0000 7fa658c00640 1 -- 172.21.15.16:0/636130723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa66001ebe0 con 0x55c796c757b0 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.484+0000 7fa6785f6740 1 -- 172.21.15.16:0/636130723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa65c0be290 con 0x55c796c757b0 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.484+0000 7fa658c00640 1 -- 172.21.15.16:0/636130723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fa66001ede0 con 0x55c796c757b0 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.484+0000 7fa6785f6740 10 client.14595 fetch_fsmap learned FSMap version 16 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.484+0000 7fa6785f6740 1 -- 172.21.15.16:0/636130723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55c796c87210 con 0x55c796c757b0 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.484+0000 7fa658c00640 1 -- 172.21.15.16:0/636130723 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa65c0be290 con 0x55c796c757b0 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.485+0000 7fa658c00640 1 -- 172.21.15.16:0/636130723 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fa66001b040 con 0x55c796c757b0 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.485+0000 7fa6785f6740 10 client.14595 fetch_fsmap finished waiting for FSMap version 16 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.485+0000 7fa6785f6740 10 client.14595 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.485+0000 7fa6785f6740 20 client.14595 populate_metadata read hostname 'smithi016' 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.485+0000 7fa6785f6740 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55c796cf7310 0x55c796cf6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.485+0000 7fa6785f6740 4 client.14595 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:31:45.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.485+0000 7fa6785f6740 1 -- 172.21.15.16:0/636130723 --> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] -- command(tid 0: {"prefix": "status"}) -- 0x55c796be93b0 con 0x55c796cf7310 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.485+0000 7fa66e000640 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55c796cf7310 0x55c796cf6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa66e000640 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55c796cf7310 0x55c796cf6020 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b95c709ce25bafb5 server_cookie=38d3bbcac64de745 in_seq=0 out_seq=0 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa658c00640 10 client.14595 ms_handle_connect on v2:172.21.15.119:6838/3722020849 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa658c00640 1 -- 172.21.15.16:0/636130723 <== mds.0 v2:172.21.15.119:6838/3722020849 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c796be93b0 con 0x55c796cf7310 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa658c00640 10 client.14595 handle_command_reply: tid=0 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa6785f6740 1 client.14595 shutdown 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa6785f6740 1 -- 172.21.15.16:0/636130723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c757b0 msgr2=0x55c796c79870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa6785f6740 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c757b0 0x55c796c79870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa66000b2b0 tx=0x7fa660014040 comp rx=0 tx=0).stop 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa6785f6740 1 -- 172.21.15.16:0/636130723 shutdown_connections 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa6785f6740 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55c796cf7310 0x55c796cf6020 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa6785f6740 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55c796c7b310 0x55c796c85950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa6785f6740 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c796c761a0 0x55c796c79db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa6785f6740 1 --2- 172.21.15.16:0/636130723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55c796c757b0 0x55c796c79870 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa658c00640 0 client.14595 ms_handle_reset on v2:172.21.15.119:6838/3722020849 2024-04-13T03:31:45.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.486+0000 7fa6785f6740 1 -- 172.21.15.16:0/636130723 >> 172.21.15.16:0/636130723 conn(0x55c796c70d20 msgr2=0x55c796c8b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.487+0000 7fa6785f6740 1 -- 172.21.15.16:0/636130723 shutdown_connections 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.487+0000 7fa6785f6740 1 -- 172.21.15.16:0/636130723 wait complete. 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.487+0000 7fa6785f6740 20 client.14595 trim_cache size 0 max 16384 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24587, 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:31:45.492 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 7.9349811880000001, 2024-04-13T03:31:45.493 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 16, 2024-04-13T03:31:45.493 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 90, 2024-04-13T03:31:45.493 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:31:45.493 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 14.310723509000001 2024-04-13T03:31:45.493 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.487+0000 7fa66fe00640 1 -- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa6540a3bb0 msgr2=0x7fa6540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.487+0000 7fa66fe00640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa6540a3bb0 0x7fa6540a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55c796c765a0 tx=0x7fa66000b410 comp rx=0 tx=0).stop 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.487+0000 7fa66fe00640 1 -- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6701096f0 msgr2=0x7fa67019ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.487+0000 7fa66fe00640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6701096f0 0x7fa67019ed60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009230 tx=0x7fa65c02c6c0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.487+0000 7fa66fe00640 1 -- 172.21.15.16:0/91538473 shutdown_connections 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.488+0000 7fa66fe00640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa6540a3bb0 0x7fa6540a6070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.488+0000 7fa66fe00640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa67010a0b0 0x7fa6701a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.488+0000 7fa66fe00640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6701096f0 0x7fa67019ed60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.488+0000 7fa66fe00640 1 --2- 172.21.15.16:0/91538473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa670108500 0x7fa67019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.488+0000 7fa66fe00640 1 -- 172.21.15.16:0/91538473 >> 172.21.15.16:0/91538473 conn(0x7fa670077490 msgr2=0x7fa67010c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.488+0000 7fa66fe00640 1 -- 172.21.15.16:0/91538473 shutdown_connections 2024-04-13T03:31:45.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:45.488+0000 7fa66fe00640 1 -- 172.21.15.16:0/91538473 wait complete. 2024-04-13T03:31:45.508 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-13T03:31:45.508 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-13T03:31:45.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph fs ls 2024-04-13T03:31:45.996 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:31:46.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:45 smithi016 ceph-mon[31772]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-13T03:31:46.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:45 smithi016 ceph-mon[31772]: Cluster is now healthy 2024-04-13T03:31:46.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:45 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4137157914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:46.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:45 smithi188 ceph-mon[36398]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-13T03:31:46.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:45 smithi188 ceph-mon[36398]: Cluster is now healthy 2024-04-13T03:31:46.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:45 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4137157914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:45 smithi119 ceph-mon[35835]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-13T03:31:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:45 smithi119 ceph-mon[35835]: Cluster is now healthy 2024-04-13T03:31:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:45 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4137157914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:47.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:46 smithi016 ceph-mon[31772]: pgmap v323: 129 pgs: 113 active+clean, 16 creating+peering; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-04-13T03:31:47.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:46 smithi188 ceph-mon[36398]: pgmap v323: 129 pgs: 113 active+clean, 16 creating+peering; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-04-13T03:31:47.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:46 smithi119 ceph-mon[35835]: pgmap v323: 129 pgs: 113 active+clean, 16 creating+peering; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-04-13T03:31:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.431+0000 7fc573200700 1 -- 172.21.15.16:0/3014169949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c108b70 msgr2=0x7fc56c108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.431+0000 7fc573200700 1 --2- 172.21.15.16:0/3014169949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c108b70 0x7fc56c108f50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009990 tx=0x7fc55c009ca0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.431+0000 7fc573200700 1 -- 172.21.15.16:0/3014169949 shutdown_connections 2024-04-13T03:31:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.431+0000 7fc573200700 1 --2- 172.21.15.16:0/3014169949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc56c1034d0 0x7fc56c10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.431+0000 7fc573200700 1 --2- 172.21.15.16:0/3014169949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c108b70 0x7fc56c108f50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.431+0000 7fc573200700 1 --2- 172.21.15.16:0/3014169949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c102b30 0x7fc56c102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.431+0000 7fc573200700 1 -- 172.21.15.16:0/3014169949 >> 172.21.15.16:0/3014169949 conn(0x7fc56c079b80 msgr2=0x7fc56c079f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.431+0000 7fc573200700 1 -- 172.21.15.16:0/3014169949 shutdown_connections 2024-04-13T03:31:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.431+0000 7fc573200700 1 -- 172.21.15.16:0/3014169949 wait complete. 2024-04-13T03:31:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.432+0000 7fc573200700 1 Processor -- start 2024-04-13T03:31:48.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.432+0000 7fc573200700 1 -- start start 2024-04-13T03:31:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc573200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c102b30 0x7fc56c074db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc573200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc56c1034d0 0x7fc56c0752f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc573200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c108b70 0x7fc56c075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc573200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c112e00 con 0x7fc56c108b70 2024-04-13T03:31:48.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc573200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c112c80 con 0x7fc56c1034d0 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc573200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c112f80 con 0x7fc56c102b30 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc571200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c108b70 0x7fc56c075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc571200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c108b70 0x7fc56c075830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53666/0 (socket says 172.21.15.16:53666) 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc571200700 1 -- 172.21.15.16:0/3001039702 learned_addr learned my addr 172.21.15.16:0/3001039702 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc56be00700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c102b30 0x7fc56c074db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.433+0000 7fc563e00700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc56c1034d0 0x7fc56c0752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.434+0000 7fc56be00700 1 -- 172.21.15.16:0/3001039702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc56c1034d0 msgr2=0x7fc56c0752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.434+0000 7fc56be00700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc56c1034d0 0x7fc56c0752f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.434+0000 7fc56be00700 1 -- 172.21.15.16:0/3001039702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c108b70 msgr2=0x7fc56c075830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.434+0000 7fc56be00700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c108b70 0x7fc56c075830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.434+0000 7fc56be00700 1 -- 172.21.15.16:0/3001039702 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c1a7790 con 0x7fc56c102b30 2024-04-13T03:31:48.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.434+0000 7fc56be00700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c102b30 0x7fc56c074db0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc55c000c00 tx=0x7fc55c00bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.435+0000 7fc569600700 1 -- 172.21.15.16:0/3001039702 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c026070 con 0x7fc56c102b30 2024-04-13T03:31:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.435+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56c1a7a20 con 0x7fc56c102b30 2024-04-13T03:31:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.435+0000 7fc569600700 1 -- 172.21.15.16:0/3001039702 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc55c003d70 con 0x7fc56c102b30 2024-04-13T03:31:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.435+0000 7fc569600700 1 -- 172.21.15.16:0/3001039702 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c00fdc0 con 0x7fc56c102b30 2024-04-13T03:31:48.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.435+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56c1a7e70 con 0x7fc56c102b30 2024-04-13T03:31:48.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.436+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56c0697a0 con 0x7fc56c102b30 2024-04-13T03:31:48.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.438+0000 7fc569600700 1 -- 172.21.15.16:0/3001039702 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc55c029070 con 0x7fc56c102b30 2024-04-13T03:31:48.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.439+0000 7fc569600700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc5500a3c40 0x7fc5500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:48.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.440+0000 7fc569600700 1 -- 172.21.15.16:0/3001039702 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc55c0d1370 con 0x7fc56c102b30 2024-04-13T03:31:48.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.440+0000 7fc563e00700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc5500a3c40 0x7fc5500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:48.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.441+0000 7fc563e00700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc5500a3c40 0x7fc5500a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc56c07a080 tx=0x7fc554009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:48.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.444+0000 7fc569600700 1 -- 172.21.15.16:0/3001039702 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc55c05dff0 con 0x7fc56c102b30 2024-04-13T03:31:48.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.728+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc56c02e040 con 0x7fc56c102b30 2024-04-13T03:31:48.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.729+0000 7fc569600700 1 -- 172.21.15.16:0/3001039702 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7fc55c013070 con 0x7fc56c102b30 2024-04-13T03:31:48.734 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-13T03:31:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.733+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc5500a3c40 msgr2=0x7fc5500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.733+0000 7fc573200700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc5500a3c40 0x7fc5500a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc56c07a080 tx=0x7fc554009260 comp rx=0 tx=0).stop 2024-04-13T03:31:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.733+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c102b30 msgr2=0x7fc56c074db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.733+0000 7fc573200700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c102b30 0x7fc56c074db0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc55c000c00 tx=0x7fc55c00bab0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.734+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 shutdown_connections 2024-04-13T03:31:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.734+0000 7fc573200700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc5500a3c40 0x7fc5500a6100 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.734+0000 7fc573200700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56c108b70 0x7fc56c075830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.734+0000 7fc573200700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc56c102b30 0x7fc56c074db0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.734+0000 7fc573200700 1 --2- 172.21.15.16:0/3001039702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc56c1034d0 0x7fc56c0752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.734+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 >> 172.21.15.16:0/3001039702 conn(0x7fc56c079b80 msgr2=0x7fc56c100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.734+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 shutdown_connections 2024-04-13T03:31:48.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:48.734+0000 7fc573200700 1 -- 172.21.15.16:0/3001039702 wait complete. 2024-04-13T03:31:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:48 smithi016 ceph-mon[31772]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-13T03:31:49.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:48 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3001039702' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-13T03:31:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:48 smithi119 ceph-mon[35835]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-13T03:31:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:48 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3001039702' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-13T03:31:49.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:48 smithi188 ceph-mon[36398]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-13T03:31:49.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:48 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3001039702' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-13T03:31:49.690 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-13T03:31:49.690 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-13T03:31:49.690 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-04-13T03:31:49.690 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-13T03:31:49.691 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-13T03:31:49.691 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-13T03:31:49.691 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-13T03:31:49.691 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-13T03:31:49.691 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-13T03:31:49.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:49.691 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-04-13T03:31:49.752 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-13T03:31:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-13T03:31:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-13T03:31:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-13T03:31:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-13T03:31:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-13T03:31:49.752 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-13T03:31:49.752 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42039sec preferred_lft 42039sec 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-04-13T03:31:49.753 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-04-13T03:31:49.759 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-13T03:31:49.759 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:31:49.759 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:31:49.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-04-13T03:31:49.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-04-13T03:31:49.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-04-13T03:31:49.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-13T03:31:49.759 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:31:49.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:31:50.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:31:50.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:50.340 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-13T03:31:50.424 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-04-13T03:31:50.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:50.428 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-04-13T03:31:50.500 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-04-13T03:31:50.500 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-13T03:31:50.500 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-13T03:31:50.500 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-13T03:31:50.500 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-13T03:31:50.501 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:31:50.501 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:31:50.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-13T03:31:50.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-13T03:31:50.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-13T03:31:50.502 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:31:50.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:31:51.313 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:50 smithi016 ceph-mon[31772]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:51.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:31:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:50 smithi119 ceph-mon[35835]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:51.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:51.329 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-13T03:31:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:50 smithi188 ceph-mon[36398]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:51.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:51.389 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-04-13T03:31:51.442 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:31:51.442 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:31:51.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-13T03:31:51.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-13T03:31:51.443 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:31:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:31:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:31:51.662 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-13T03:31:51.662 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:31:51.662 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:31:51.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-13T03:31:51.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-13T03:31:51.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-13T03:31:51.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-13T03:31:51.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-13T03:31:51.663 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:31:51.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:31:52.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:31:52.038 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:31:52.038 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:31:52.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-04-13T03:31:52.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-04-13T03:31:52.038 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:31:52.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:31:52.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:31:52.279 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-13T03:31:52.279 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-13T03:31:52.279 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-13T03:31:52.325 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-13T03:31:52.325 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-13T03:31:52.325 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-13T03:31:52.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-13T03:31:52.523 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-13T03:31:52.577 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-13T03:31:52.577 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-13T03:31:52.578 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b14a79184c4d09fa24bb5ad32d63f02f1ac60104185edcd72e074ae50e4a9cdb/merged 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/dfe0523c7ad7eea7e91dfef79acdb489fb563758c7e2be0355f3c5fde356bbe1/merged 2024-04-13T03:31:52.579 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/e2195d89eb89a1d50aa716437158def2cf582a6498de1d8f662e7296ef688b25/merged 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/977a267fa65a868992109d51245c7012be5c7adbebe6caf285e03c2597822c8f/merged 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/4d2c8c810d9b9d160b24faf67d4dbe3833a2023f4d8d13a624e2155e99a974ab/merged 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c5eb6a7470eec26797b86c0381da18c067fdf67c0dc91a8c871e7412c02fcf36/merged 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/1edf347a4216c28dfdf719fba3e02c262cf0de72e0eb8ae509203b17e594bd03/merged 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/566a08aa1639e3f8913bd690f124b9e90b16f9a839302e5824a07ca451e0f8cd/merged 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d276ce87afd6bc62073989fbc53ba8a9daa74e00480d1644522b10c24604b262/merged 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/03570dcc079068bf3b2967deb20b28f695a3265f4c95125b5593a025ff86c2a6/merged 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-13T03:31:52.580 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-13T03:31:52.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:52.581 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-13T03:31:52.635 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-13T03:31:52.635 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-13T03:31:52.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-13T03:31:52.718 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-13T03:31:52.760 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-13T03:31:52.760 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-13T03:31:52.760 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-13T03:31:52.760 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-13T03:31:52.760 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-13T03:31:52.760 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-13T03:31:52.760 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-13T03:31:52.760 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-13T03:31:52.761 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b14a79184c4d09fa24bb5ad32d63f02f1ac60104185edcd72e074ae50e4a9cdb/merged 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/dfe0523c7ad7eea7e91dfef79acdb489fb563758c7e2be0355f3c5fde356bbe1/merged 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/e2195d89eb89a1d50aa716437158def2cf582a6498de1d8f662e7296ef688b25/merged 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/977a267fa65a868992109d51245c7012be5c7adbebe6caf285e03c2597822c8f/merged 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/4d2c8c810d9b9d160b24faf67d4dbe3833a2023f4d8d13a624e2155e99a974ab/merged 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c5eb6a7470eec26797b86c0381da18c067fdf67c0dc91a8c871e7412c02fcf36/merged 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/1edf347a4216c28dfdf719fba3e02c262cf0de72e0eb8ae509203b17e594bd03/merged 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/566a08aa1639e3f8913bd690f124b9e90b16f9a839302e5824a07ca451e0f8cd/merged 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d276ce87afd6bc62073989fbc53ba8a9daa74e00480d1644522b10c24604b262/merged 2024-04-13T03:31:52.762 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/03570dcc079068bf3b2967deb20b28f695a3265f4c95125b5593a025ff86c2a6/merged 2024-04-13T03:31:52.763 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-13T03:31:52.763 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-13T03:31:52.763 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-13T03:31:52.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:52.764 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-13T03:31:52.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:2024-04-13T03:31:52.850+0000 7fb360733480 -1 init, newargv = 0x5619bd073070 newargc=15 2024-04-13T03:31:52.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[72692]: starting ceph client 2024-04-13T03:31:52.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[72692]: starting fuse 2024-04-13T03:31:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:52 smithi119 ceph-mon[35835]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:52 smithi016 ceph-mon[31772]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:52 smithi188 ceph-mon[36398]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:53.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-13T03:31:54.071 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-13T03:31:54.132 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-13T03:31:54.132 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-13T03:31:54.133 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/b14a79184c4d09fa24bb5ad32d63f02f1ac60104185edcd72e074ae50e4a9cdb/merged 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/dfe0523c7ad7eea7e91dfef79acdb489fb563758c7e2be0355f3c5fde356bbe1/merged 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/0 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-13T03:31:54.134 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/e2195d89eb89a1d50aa716437158def2cf582a6498de1d8f662e7296ef688b25/merged 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/977a267fa65a868992109d51245c7012be5c7adbebe6caf285e03c2597822c8f/merged 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/4d2c8c810d9b9d160b24faf67d4dbe3833a2023f4d8d13a624e2155e99a974ab/merged 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/c5eb6a7470eec26797b86c0381da18c067fdf67c0dc91a8c871e7412c02fcf36/merged 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/1edf347a4216c28dfdf719fba3e02c262cf0de72e0eb8ae509203b17e594bd03/merged 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/566a08aa1639e3f8913bd690f124b9e90b16f9a839302e5824a07ca451e0f8cd/merged 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/d276ce87afd6bc62073989fbc53ba8a9daa74e00480d1644522b10c24604b262/merged 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/containers/storage/overlay/03570dcc079068bf3b2967deb20b28f695a3265f4c95125b5593a025ff86c2a6/merged 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-13T03:31:54.135 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-13T03:31:54.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:54.136 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-13T03:31:54.192 INFO:teuthology.orchestra.run.smithi016.stdout:119 2024-04-13T03:31:54.192 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-04-13T03:31:54.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-04-13T03:31:54.192 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-04-13T03:31:54.192 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-04-13T03:31:54.192 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-13T03:31:54.193 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-13T03:31:54.194 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-04-13T03:31:54.194 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-04-13T03:31:54.194 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-04-13T03:31:54.194 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-13T03:31:54.194 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-13T03:31:54.194 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.admin")) 2024-04-13T03:31:54.194 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-04-13T03:31:54.383 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.admin.72692.asok 2024-04-13T03:31:54.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:31:54.403 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72692.asok 2024-04-13T03:31:54.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:54.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72692.asok status 2024-04-13T03:31:54.644 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "218a691417ef57c5d22a8d5cf3c5f41f7b542838", 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)", 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "admin", 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "72692", 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24691, 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-04-13T03:31:54.645 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 24691 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4199784585 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4199784585 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:31:54.646 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.24691 172.21.15.16:0/4199784585", 2024-04-13T03:31:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "172.21.15.16:0/4199784585", 2024-04-13T03:31:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-04-13T03:31:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 16, 2024-04-13T03:31:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 90, 2024-04-13T03:31:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-04-13T03:31:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-04-13T03:31:54.647 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-04-13T03:31:54.647 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:31:54.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:54.661 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-04-13T03:31:54.661 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-13T03:31:54.733 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-04-13T03:31:54.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-13T03:31:54.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:31:54.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-13T03:31:54.842 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-13T03:31:54.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-13T03:31:54.986 INFO:teuthology.orchestra.run.smithi016.stdout:# file: . 2024-04-13T03:31:54.986 INFO:teuthology.orchestra.run.smithi016.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-13T03:31:54.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:54.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:31:54.995 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:31:54.996 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-13T03:31:55.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:31:55.191 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:54 smithi016 ceph-mon[31772]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:54 smithi119 ceph-mon[35835]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.328+0000 7f18b7e00640 1 -- 172.21.15.16:0/3196200023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8104040 msgr2=0x7f18b8104460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.328+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3196200023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8104040 0x7f18b8104460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009ca0 tx=0x7f18a802c0f0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.328+0000 7f18b7e00640 1 -- 172.21.15.16:0/3196200023 shutdown_connections 2024-04-13T03:31:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.328+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3196200023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8105bf0 0x7f18b810c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.328+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3196200023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b8105230 0x7f18b81056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.328+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3196200023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8104040 0x7f18b8104460 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.328+0000 7f18b7e00640 1 -- 172.21.15.16:0/3196200023 >> 172.21.15.16:0/3196200023 conn(0x7f18b80ff5d0 msgr2=0x7f18b81019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.328+0000 7f18b7e00640 1 -- 172.21.15.16:0/3196200023 shutdown_connections 2024-04-13T03:31:55.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.328+0000 7f18b7e00640 1 -- 172.21.15.16:0/3196200023 wait complete. 2024-04-13T03:31:55.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.329+0000 7f18b7e00640 1 Processor -- start 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.329+0000 7f18b7e00640 1 -- start start 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18b7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b8104040 0x7f18b819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18b7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8105230 0x7f18b819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18b7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8105bf0 0x7f18b81a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18b7e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810fcd0 con 0x7f18b8104040 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18b7e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810fb50 con 0x7f18b8105bf0 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18b7e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810fe50 con 0x7f18b8105230 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18b6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b8104040 0x7f18b819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18ae000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8105230 0x7f18b819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18ae000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8105230 0x7f18b819a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:38236/0 (socket says 172.21.15.16:38236) 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18b7400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8105bf0 0x7f18b81a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.330+0000 7f18ae000640 1 -- 172.21.15.16:0/3961265051 learned_addr learned my addr 172.21.15.16:0/3961265051 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:55.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18ae000640 1 -- 172.21.15.16:0/3961265051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8105bf0 msgr2=0x7f18b81a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18ae000640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8105bf0 0x7f18b81a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18ae000640 1 -- 172.21.15.16:0/3961265051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b8104040 msgr2=0x7f18b819a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18ae000640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b8104040 0x7f18b819a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18ae000640 1 -- 172.21.15.16:0/3961265051 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b81a1fb0 con 0x7f18b8105230 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18b7400640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8105bf0 0x7f18b81a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18ae000640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8105230 0x7f18b819a8d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f18a000e9f0 tx=0x7f18a000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18afe00640 1 -- 172.21.15.16:0/3961265051 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a000cd80 con 0x7f18b8105230 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18afe00640 1 -- 172.21.15.16:0/3961265051 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a0010910 con 0x7f18b8105230 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18afe00640 1 -- 172.21.15.16:0/3961265051 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0018990 con 0x7f18b8105230 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.331+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b81a22a0 con 0x7f18b8105230 2024-04-13T03:31:55.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.332+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b81a2840 con 0x7f18b8105230 2024-04-13T03:31:55.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.333+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b810b380 con 0x7f18b8105230 2024-04-13T03:31:55.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.335+0000 7f18afe00640 1 -- 172.21.15.16:0/3961265051 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f18a0010ac0 con 0x7f18b8105230 2024-04-13T03:31:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.336+0000 7f18afe00640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f189c0a3c30 0x7f189c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.336+0000 7f18afe00640 1 -- 172.21.15.16:0/3961265051 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f18a00d0d10 con 0x7f18b8105230 2024-04-13T03:31:55.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.337+0000 7f18b6a00640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f189c0a3c30 0x7f189c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:55.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.337+0000 7f18b6a00640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f189c0a3c30 0x7f189c0a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f18a802c630 tx=0x7f18a8009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:55.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.338+0000 7f18afe00640 1 -- 172.21.15.16:0/3961265051 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18a009ce00 con 0x7f18b8105230 2024-04-13T03:31:55.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:54 smithi188 ceph-mon[36398]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.545+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18b802d830 con 0x7f18b8105230 2024-04-13T03:31:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.548+0000 7f18afe00640 1 -- 172.21.15.16:0/3961265051 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7f18a00a1100 con 0x7f18b8105230 2024-04-13T03:31:55.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:55.550 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:31:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f189c0a3c30 msgr2=0x7f189c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f189c0a3c30 0x7f189c0a60f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f18a802c630 tx=0x7f18a8009360 comp rx=0 tx=0).stop 2024-04-13T03:31:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8105230 msgr2=0x7f18b819a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8105230 0x7f18b819a8d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f18a000e9f0 tx=0x7f18a000eeb0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 shutdown_connections 2024-04-13T03:31:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f189c0a3c30 0x7f189c0a60f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8105bf0 0x7f18b81a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18b8105230 0x7f18b819a8d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 --2- 172.21.15.16:0/3961265051 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b8104040 0x7f18b819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 >> 172.21.15.16:0/3961265051 conn(0x7f18b80ff5d0 msgr2=0x7f18b8108550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 shutdown_connections 2024-04-13T03:31:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.551+0000 7f18b7e00640 1 -- 172.21.15.16:0/3961265051 wait complete. 2024-04-13T03:31:55.572 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-13T03:31:34.802688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-13T03:31:36.036786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-13T03:31:39.824413+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-13T03:31:55.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:31:55.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.802+0000 7f616b200640 1 -- 172.21.15.16:0/2503473280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f616c1052d0 msgr2=0x7f616c1076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:55.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.802+0000 7f616b200640 1 --2- 172.21.15.16:0/2503473280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f616c1052d0 0x7f616c1076c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6158009900 tx=0x7f615802bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.802+0000 7f616b200640 1 -- 172.21.15.16:0/2503473280 shutdown_connections 2024-04-13T03:31:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.802+0000 7f616b200640 1 --2- 172.21.15.16:0/2503473280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c107c00 0x7f616c10a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.802+0000 7f616b200640 1 --2- 172.21.15.16:0/2503473280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f616c1052d0 0x7f616c1076c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.802+0000 7f616b200640 1 --2- 172.21.15.16:0/2503473280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f616c1029a0 0x7f616c104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.802+0000 7f616b200640 1 -- 172.21.15.16:0/2503473280 >> 172.21.15.16:0/2503473280 conn(0x7f616c0fc390 msgr2=0x7f616c0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.802+0000 7f616b200640 1 -- 172.21.15.16:0/2503473280 shutdown_connections 2024-04-13T03:31:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.802+0000 7f616b200640 1 -- 172.21.15.16:0/2503473280 wait complete. 2024-04-13T03:31:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.803+0000 7f616b200640 1 Processor -- start 2024-04-13T03:31:55.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.804+0000 7f616b200640 1 -- start start 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.804+0000 7f616b200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f616c1029a0 0x7f616c19a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.804+0000 7f616b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c1052d0 0x7f616c19a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.804+0000 7f616b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f616c107c00 0x7f616c1a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.804+0000 7f616b200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f616c10db90 con 0x7f616c1029a0 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.804+0000 7f616b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f616c10da10 con 0x7f616c107c00 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.804+0000 7f616b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f616c10dd10 con 0x7f616c1052d0 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.804+0000 7f6169e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f616c1029a0 0x7f616c19a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.804+0000 7f6169400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c1052d0 0x7f616c19a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f6169400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c1052d0 0x7f616c19a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:38250/0 (socket says 172.21.15.16:38250) 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f6169e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f616c1029a0 0x7f616c19a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56848/0 (socket says 172.21.15.16:56848) 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f6169400640 1 -- 172.21.15.16:0/3785116292 learned_addr learned my addr 172.21.15.16:0/3785116292 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f616a800640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f616c107c00 0x7f616c1a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f6169400640 1 -- 172.21.15.16:0/3785116292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f616c107c00 msgr2=0x7f616c1a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f6169400640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f616c107c00 0x7f616c1a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f6169400640 1 -- 172.21.15.16:0/3785116292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f616c1029a0 msgr2=0x7f616c19a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f6169400640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f616c1029a0 0x7f616c19a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f6169400640 1 -- 172.21.15.16:0/3785116292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616c1a2060 con 0x7f616c1052d0 2024-04-13T03:31:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.805+0000 7f6169400640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c1052d0 0x7f616c19a980 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6158000c00 tx=0x7f615802c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.806+0000 7f6162a00640 1 -- 172.21.15.16:0/3785116292 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158041070 con 0x7f616c1052d0 2024-04-13T03:31:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.806+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616c1a22f0 con 0x7f616c1052d0 2024-04-13T03:31:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.806+0000 7f6162a00640 1 -- 172.21.15.16:0/3785116292 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6158004450 con 0x7f616c1052d0 2024-04-13T03:31:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.806+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616c1a2830 con 0x7f616c1052d0 2024-04-13T03:31:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.806+0000 7f6162a00640 1 -- 172.21.15.16:0/3785116292 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158034440 con 0x7f616c1052d0 2024-04-13T03:31:55.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.807+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f616c109750 con 0x7f616c1052d0 2024-04-13T03:31:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.809+0000 7f6162a00640 1 -- 172.21.15.16:0/3785116292 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6158044070 con 0x7f616c1052d0 2024-04-13T03:31:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.810+0000 7f6162a00640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f61500a3bb0 0x7f61500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:55.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.811+0000 7f6162a00640 1 -- 172.21.15.16:0/3785116292 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f61580f2360 con 0x7f616c1052d0 2024-04-13T03:31:55.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.811+0000 7f6169e00640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f61500a3bb0 0x7f61500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:55.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.811+0000 7f6169e00640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f61500a3bb0 0x7f61500a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f615c005fd0 tx=0x7f615c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:55.812+0000 7f6162a00640 1 -- 172.21.15.16:0/3785116292 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61580be4e0 con 0x7f616c1052d0 2024-04-13T03:31:56.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.019+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f616c02d5e0 con 0x7f616c1052d0 2024-04-13T03:31:56.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.020+0000 7f6162a00640 1 -- 172.21.15.16:0/3785116292 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f61580c27e0 con 0x7f616c1052d0 2024-04-13T03:31:56.022 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f61500a3bb0 msgr2=0x7f61500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f61500a3bb0 0x7f61500a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f615c005fd0 tx=0x7f615c009500 comp rx=0 tx=0).stop 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c1052d0 msgr2=0x7f616c19a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c1052d0 0x7f616c19a980 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6158000c00 tx=0x7f615802c640 comp rx=0 tx=0).stop 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 shutdown_connections 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f61500a3bb0 0x7f61500a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f616c107c00 0x7f616c1a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c1052d0 0x7f616c19a980 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 --2- 172.21.15.16:0/3785116292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f616c1029a0 0x7f616c19a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 >> 172.21.15.16:0/3785116292 conn(0x7f616c0fc390 msgr2=0x7f616c102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:56.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 shutdown_connections 2024-04-13T03:31:56.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.024+0000 7f616b200640 1 -- 172.21.15.16:0/3785116292 wait complete. 2024-04-13T03:31:56.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-13T03:31:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.278+0000 7f8b11a00640 1 -- 172.21.15.16:0/1483584676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c107bd0 msgr2=0x7f8b0c109fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.278+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1483584676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c107bd0 0x7f8b0c109fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009be0 tx=0x7f8b0402bef0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.278+0000 7f8b11a00640 1 -- 172.21.15.16:0/1483584676 shutdown_connections 2024-04-13T03:31:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.278+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1483584676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c107bd0 0x7f8b0c109fc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.278+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1483584676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b0c100420 0x7f8b0c107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.278+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1483584676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b0c0ffac0 0x7f8b0c0ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.278+0000 7f8b11a00640 1 -- 172.21.15.16:0/1483584676 >> 172.21.15.16:0/1483584676 conn(0x7f8b0c0fb680 msgr2=0x7f8b0c0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.278+0000 7f8b11a00640 1 -- 172.21.15.16:0/1483584676 shutdown_connections 2024-04-13T03:31:56.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.278+0000 7f8b11a00640 1 -- 172.21.15.16:0/1483584676 wait complete. 2024-04-13T03:31:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.280+0000 7f8b11a00640 1 Processor -- start 2024-04-13T03:31:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.280+0000 7f8b11a00640 1 -- start start 2024-04-13T03:31:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.280+0000 7f8b11a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c0ffac0 0x7f8b0c1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.280+0000 7f8b11a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b0c100420 0x7f8b0c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.280+0000 7f8b11a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b0c107bd0 0x7f8b0c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:56.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.280+0000 7f8b11a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0c10db90 con 0x7f8b0c100420 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.280+0000 7f8b11a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0c10da10 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.280+0000 7f8b11a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0c10dd10 con 0x7f8b0c107bd0 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b0c100420 0x7f8b0c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c0ffac0 0x7f8b0c1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c0ffac0 0x7f8b0c1981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:55458/0 (socket says 172.21.15.16:55458) 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0be00640 1 -- 172.21.15.16:0/1147012838 learned_addr learned my addr 172.21.15.16:0/1147012838 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b11000640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b0c107bd0 0x7f8b0c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0be00640 1 -- 172.21.15.16:0/1147012838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b0c107bd0 msgr2=0x7f8b0c19f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0be00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b0c107bd0 0x7f8b0c19f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0be00640 1 -- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b0c100420 msgr2=0x7f8b0c198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0be00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b0c100420 0x7f8b0c198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0be00640 1 -- 172.21.15.16:0/1147012838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0c19fdf0 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0b400640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b0c100420 0x7f8b0c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:56.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.281+0000 7f8b0be00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c0ffac0 0x7f8b0c1981d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00e9f0 tx=0x7f8afc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:56.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.282+0000 7f8b08c00640 1 -- 172.21.15.16:0/1147012838 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc00ccc0 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.282+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b0c1a00e0 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.282+0000 7f8b08c00640 1 -- 172.21.15.16:0/1147012838 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8afc010910 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.282+0000 7f8b08c00640 1 -- 172.21.15.16:0/1147012838 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc0188a0 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.282+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b0c1a0680 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.283+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b0c1064a0 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.285+0000 7f8b08c00640 1 -- 172.21.15.16:0/1147012838 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8afc010ac0 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.286+0000 7f8b08c00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8af00a3bb0 0x7f8af00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.287+0000 7f8b08c00640 1 -- 172.21.15.16:0/1147012838 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8afc0d0ad0 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.287+0000 7f8b0b400640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8af00a3bb0 0x7f8af00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:56.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.288+0000 7f8b0b400640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8af00a3bb0 0x7f8af00a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c1996e0 tx=0x7f8af4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:56.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.291+0000 7f8b08c00640 1 -- 172.21.15.16:0/1147012838 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8afc09d210 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:55 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3961265051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:56.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:55 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3961265051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:56.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:55 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3961265051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:56.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.505+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b0c062bc0 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.506+0000 7f8b08c00640 1 -- 172.21.15.16:0/1147012838 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f8afc09cb10 con 0x7f8b0c0ffac0 2024-04-13T03:31:56.507 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:31:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.508+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8af00a3bb0 msgr2=0x7f8af00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.508+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8af00a3bb0 0x7f8af00a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c1996e0 tx=0x7f8af4009260 comp rx=0 tx=0).stop 2024-04-13T03:31:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.508+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c0ffac0 msgr2=0x7f8b0c1981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.508+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c0ffac0 0x7f8b0c1981d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00e9f0 tx=0x7f8afc00edf0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.509+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 shutdown_connections 2024-04-13T03:31:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.509+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8af00a3bb0 0x7f8af00a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.509+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b0c107bd0 0x7f8b0c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.509+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b0c100420 0x7f8b0c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.509+0000 7f8b11a00640 1 --2- 172.21.15.16:0/1147012838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b0c0ffac0 0x7f8b0c1981d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.509+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 >> 172.21.15.16:0/1147012838 conn(0x7f8b0c0fb680 msgr2=0x7f8b0c103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.509+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 shutdown_connections 2024-04-13T03:31:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.509+0000 7f8b11a00640 1 -- 172.21.15.16:0/1147012838 wait complete. 2024-04-13T03:31:56.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-13T03:31:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.753+0000 7f8a05200640 1 -- 172.21.15.16:0/2389284084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00107c50 msgr2=0x7f8a0010a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.753+0000 7f8a05200640 1 --2- 172.21.15.16:0/2389284084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00107c50 0x7f8a0010a040 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009ca0 tx=0x7f89f802c100 comp rx=0 tx=0).stop 2024-04-13T03:31:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.753+0000 7f8a05200640 1 -- 172.21.15.16:0/2389284084 shutdown_connections 2024-04-13T03:31:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.753+0000 7f8a05200640 1 --2- 172.21.15.16:0/2389284084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00107c50 0x7f8a0010a040 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.753+0000 7f8a05200640 1 --2- 172.21.15.16:0/2389284084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00105320 0x7f8a00107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.753+0000 7f8a05200640 1 --2- 172.21.15.16:0/2389284084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001029f0 0x7f8a00104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.753+0000 7f8a05200640 1 -- 172.21.15.16:0/2389284084 >> 172.21.15.16:0/2389284084 conn(0x7f8a000fc360 msgr2=0x7f8a000fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.754+0000 7f8a05200640 1 -- 172.21.15.16:0/2389284084 shutdown_connections 2024-04-13T03:31:56.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.754+0000 7f8a05200640 1 -- 172.21.15.16:0/2389284084 wait complete. 2024-04-13T03:31:56.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.755+0000 7f8a05200640 1 Processor -- start 2024-04-13T03:31:56.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.755+0000 7f8a05200640 1 -- start start 2024-04-13T03:31:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.755+0000 7f8a05200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001029f0 0x7f8a00102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.755+0000 7f8a05200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00105320 0x7f8a001009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.755+0000 7f8a05200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00107c50 0x7f8a00100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.755+0000 7f8a05200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0010db40 con 0x7f8a001029f0 2024-04-13T03:31:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.755+0000 7f8a05200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0010d9c0 con 0x7f8a00105320 2024-04-13T03:31:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.755+0000 7f8a05200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0010dcc0 con 0x7f8a00107c50 2024-04-13T03:31:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89ff400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001029f0 0x7f8a00102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:56.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89ff400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001029f0 0x7f8a00102360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56890/0 (socket says 172.21.15.16:56890) 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89ff400640 1 -- 172.21.15.16:0/1133464047 learned_addr learned my addr 172.21.15.16:0/1133464047 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89f6a00640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00105320 0x7f8a001009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89ffe00640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00107c50 0x7f8a00100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89ff400640 1 -- 172.21.15.16:0/1133464047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00107c50 msgr2=0x7f8a00100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89ff400640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00107c50 0x7f8a00100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89ff400640 1 -- 172.21.15.16:0/1133464047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00105320 msgr2=0x7f8a001009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89ff400640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00105320 0x7f8a001009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.756+0000 7f89ff400640 1 -- 172.21.15.16:0/1133464047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a00101770 con 0x7f8a001029f0 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.757+0000 7f89ff400640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001029f0 0x7f8a00102360 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f89f000ea60 tx=0x7f89f000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.757+0000 7f89fcc00640 1 -- 172.21.15.16:0/1133464047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f000cde0 con 0x7f8a001029f0 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.757+0000 7f89fcc00640 1 -- 172.21.15.16:0/1133464047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f00035f0 con 0x7f8a001029f0 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.757+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a001a22b0 con 0x7f8a001029f0 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.757+0000 7f89fcc00640 1 -- 172.21.15.16:0/1133464047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0010670 con 0x7f8a001029f0 2024-04-13T03:31:56.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.757+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a001a2850 con 0x7f8a001029f0 2024-04-13T03:31:56.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.758+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a000fde50 con 0x7f8a001029f0 2024-04-13T03:31:56.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.760+0000 7f89fcc00640 1 -- 172.21.15.16:0/1133464047 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f89f00037a0 con 0x7f8a001029f0 2024-04-13T03:31:56.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.762+0000 7f89fcc00640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f89e40a3c30 0x7f89e40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:56.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.762+0000 7f89fcc00640 1 -- 172.21.15.16:0/1133464047 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f89f00d0930 con 0x7f8a001029f0 2024-04-13T03:31:56.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.762+0000 7f89f6a00640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f89e40a3c30 0x7f89e40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:56.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.763+0000 7f89f6a00640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f89e40a3c30 0x7f89e40a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8a001020d0 tx=0x7f89e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:56.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.767+0000 7f89fcc00640 1 -- 172.21.15.16:0/1133464047 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89f009d070 con 0x7f8a001029f0 2024-04-13T03:31:56.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.979+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a00062bc0 con 0x7f8a001029f0 2024-04-13T03:31:56.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.980+0000 7f89fcc00640 1 -- 172.21.15.16:0/1133464047 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f89f009c970 con 0x7f8a001029f0 2024-04-13T03:31:56.982 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f89e40a3c30 msgr2=0x7f89e40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f89e40a3c30 0x7f89e40a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8a001020d0 tx=0x7f89e8005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001029f0 msgr2=0x7f8a00102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001029f0 0x7f8a00102360 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f89f000ea60 tx=0x7f89f000ef20 comp rx=0 tx=0).stop 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 shutdown_connections 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f89e40a3c30 0x7f89e40a60f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00107c50 0x7f8a00100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00105320 0x7f8a001009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 --2- 172.21.15.16:0/1133464047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001029f0 0x7f8a00102360 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.983+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 >> 172.21.15.16:0/1133464047 conn(0x7f8a000fc360 msgr2=0x7f8a00103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.984+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 shutdown_connections 2024-04-13T03:31:56.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:56.984+0000 7f8a05200640 1 -- 172.21.15.16:0/1133464047 wait complete. 2024-04-13T03:31:57.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-13T03:31:57.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.232+0000 7fb0d5a00640 1 -- 172.21.15.16:0/3570543992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d010a1d0 msgr2=0x7fb0d0110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.232+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/3570543992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d010a1d0 0x7fb0d0110a60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009ca0 tx=0x7fb0c802c100 comp rx=0 tx=0).stop 2024-04-13T03:31:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.232+0000 7fb0d5a00640 1 -- 172.21.15.16:0/3570543992 shutdown_connections 2024-04-13T03:31:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.232+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/3570543992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d010a1d0 0x7fb0d0110a60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.232+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/3570543992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d0109810 0x7fb0d0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.232+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/3570543992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d0108620 0x7fb0d0108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.232+0000 7fb0d5a00640 1 -- 172.21.15.16:0/3570543992 >> 172.21.15.16:0/3570543992 conn(0x7fb0d00775d0 msgr2=0x7fb0d00799f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.232+0000 7fb0d5a00640 1 -- 172.21.15.16:0/3570543992 shutdown_connections 2024-04-13T03:31:57.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.232+0000 7fb0d5a00640 1 -- 172.21.15.16:0/3570543992 wait complete. 2024-04-13T03:31:57.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.233+0000 7fb0d5a00640 1 Processor -- start 2024-04-13T03:31:57.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.233+0000 7fb0d5a00640 1 -- start start 2024-04-13T03:31:57.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0d5a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d0108620 0x7fb0d019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:57.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0d5a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d0109810 0x7fb0d019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:57.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0d5a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d010a1d0 0x7fb0d01a5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:57.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0d5a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0114280 con 0x7fb0d0109810 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0d5a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0114100 con 0x7fb0d0108620 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0d5a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0114400 con 0x7fb0d010a1d0 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0ce000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d0109810 0x7fb0d019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0ce000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d0109810 0x7fb0d019ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56916/0 (socket says 172.21.15.16:56916) 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0ce000640 1 -- 172.21.15.16:0/2475233279 learned_addr learned my addr 172.21.15.16:0/2475233279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0cea00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d0108620 0x7fb0d019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0cf400640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d010a1d0 0x7fb0d01a5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0ce000640 1 -- 172.21.15.16:0/2475233279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d010a1d0 msgr2=0x7fb0d01a5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0ce000640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d010a1d0 0x7fb0d01a5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0ce000640 1 -- 172.21.15.16:0/2475233279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d0108620 msgr2=0x7fb0d019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0ce000640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d0108620 0x7fb0d019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0ce000640 1 -- 172.21.15.16:0/2475233279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d01a6510 con 0x7fb0d0109810 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0cf400640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d010a1d0 0x7fb0d01a5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:57.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.234+0000 7fb0cea00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d0108620 0x7fb0d019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.235+0000 7fb0ce000640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d0109810 0x7fb0d019ee30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00e9f0 tx=0x7fb0bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.235+0000 7fb0c7400640 1 -- 172.21.15.16:0/2475233279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc00cd80 con 0x7fb0d0109810 2024-04-13T03:31:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.235+0000 7fb0c7400640 1 -- 172.21.15.16:0/2475233279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0bc010910 con 0x7fb0d0109810 2024-04-13T03:31:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.235+0000 7fb0c7400640 1 -- 172.21.15.16:0/2475233279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc018990 con 0x7fb0d0109810 2024-04-13T03:31:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.235+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d01a6800 con 0x7fb0d0109810 2024-04-13T03:31:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.235+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d01a6da0 con 0x7fb0d0109810 2024-04-13T03:31:57.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.236+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d010f940 con 0x7fb0d0109810 2024-04-13T03:31:57.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.239+0000 7fb0c7400640 1 -- 172.21.15.16:0/2475233279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb0bc010ac0 con 0x7fb0d0109810 2024-04-13T03:31:57.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.241+0000 7fb0c7400640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0b40a3c30 0x7fb0b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.241+0000 7fb0c7400640 1 -- 172.21.15.16:0/2475233279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb0bc0d08d0 con 0x7fb0d0109810 2024-04-13T03:31:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.241+0000 7fb0cea00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0b40a3c30 0x7fb0b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:57.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.242+0000 7fb0cea00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0b40a3c30 0x7fb0b40a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0109670 tx=0x7fb0c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.248+0000 7fb0c7400640 1 -- 172.21.15.16:0/2475233279 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0bc09ca20 con 0x7fb0d0109810 2024-04-13T03:31:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:56 smithi119 ceph-mon[35835]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-13T03:31:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:56 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3785116292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:31:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:56 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1147012838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:31:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:56 smithi016 ceph-mon[31772]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-13T03:31:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:56 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3785116292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:31:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:56 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1147012838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:31:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:56 smithi188 ceph-mon[36398]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-13T03:31:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:56 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3785116292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:31:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:56 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1147012838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:31:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.459+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb0d0062bc0 con 0x7fb0d0109810 2024-04-13T03:31:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.460+0000 7fb0c7400640 1 -- 172.21.15.16:0/2475233279 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fb0bc0a0d20 con 0x7fb0d0109810 2024-04-13T03:31:57.461 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0b40a3c30 msgr2=0x7fb0b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0b40a3c30 0x7fb0b40a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0109670 tx=0x7fb0c0005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d0109810 msgr2=0x7fb0d019ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d0109810 0x7fb0d019ee30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00e9f0 tx=0x7fb0bc00eeb0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 shutdown_connections 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0b40a3c30 0x7fb0b40a60f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0d010a1d0 0x7fb0d01a5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0d0109810 0x7fb0d019ee30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 --2- 172.21.15.16:0/2475233279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d0108620 0x7fb0d019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.463+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 >> 172.21.15.16:0/2475233279 conn(0x7fb0d00775d0 msgr2=0x7fb0d010cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.464+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 shutdown_connections 2024-04-13T03:31:57.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.464+0000 7fb0d5a00640 1 -- 172.21.15.16:0/2475233279 wait complete. 2024-04-13T03:31:57.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-13T03:31:57.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.714+0000 7fcf25000640 1 -- 172.21.15.16:0/3670929720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 msgr2=0x7fcf2010c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.714+0000 7fcf25000640 1 --2- 172.21.15.16:0/3670929720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 0x7fcf2010c680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009ca0 tx=0x7fcf1802c0f0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.714+0000 7fcf25000640 1 -- 172.21.15.16:0/3670929720 shutdown_connections 2024-04-13T03:31:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.714+0000 7fcf25000640 1 --2- 172.21.15.16:0/3670929720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 0x7fcf2010c680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.714+0000 7fcf25000640 1 --2- 172.21.15.16:0/3670929720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf20105430 0x7fcf201058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.714+0000 7fcf25000640 1 --2- 172.21.15.16:0/3670929720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf20104240 0x7fcf20104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.714+0000 7fcf25000640 1 -- 172.21.15.16:0/3670929720 >> 172.21.15.16:0/3670929720 conn(0x7fcf200ff7b0 msgr2=0x7fcf20101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.714+0000 7fcf25000640 1 -- 172.21.15.16:0/3670929720 shutdown_connections 2024-04-13T03:31:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.715+0000 7fcf25000640 1 -- 172.21.15.16:0/3670929720 wait complete. 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.715+0000 7fcf25000640 1 Processor -- start 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf25000640 1 -- start start 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf25000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf20104240 0x7fcf2019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf25000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf20105430 0x7fcf2019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf25000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 0x7fcf201a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf25000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2010fed0 con 0x7fcf20105df0 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf25000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2010fd50 con 0x7fcf20104240 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf25000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcf20110050 con 0x7fcf20105430 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf1e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 0x7fcf201a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf1e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 0x7fcf201a1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56928/0 (socket says 172.21.15.16:56928) 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf15400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf20105430 0x7fcf2019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:57.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf1e800640 1 -- 172.21.15.16:0/1222148286 learned_addr learned my addr 172.21.15.16:0/1222148286 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.716+0000 7fcf1de00640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf20104240 0x7fcf2019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf1e800640 1 -- 172.21.15.16:0/1222148286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf20105430 msgr2=0x7fcf2019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf1e800640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf20105430 0x7fcf2019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf1e800640 1 -- 172.21.15.16:0/1222148286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf20104240 msgr2=0x7fcf2019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf1e800640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf20104240 0x7fcf2019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf1e800640 1 -- 172.21.15.16:0/1222148286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf201a20e0 con 0x7fcf20105df0 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf1e800640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 0x7fcf201a1a70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcf1802c630 tx=0x7fcf18032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf17400640 1 -- 172.21.15.16:0/1222148286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf18041020 con 0x7fcf20105df0 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf17400640 1 -- 172.21.15.16:0/1222148286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf18034cd0 con 0x7fcf20105df0 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf201a2370 con 0x7fcf20105df0 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.717+0000 7fcf17400640 1 -- 172.21.15.16:0/1222148286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf18039040 con 0x7fcf20105df0 2024-04-13T03:31:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.718+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf201a2860 con 0x7fcf20105df0 2024-04-13T03:31:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.719+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf2010b560 con 0x7fcf20105df0 2024-04-13T03:31:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.721+0000 7fcf17400640 1 -- 172.21.15.16:0/1222148286 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcf18044020 con 0x7fcf20105df0 2024-04-13T03:31:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.722+0000 7fcf17400640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf040a3bb0 0x7fcf040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.722+0000 7fcf17400640 1 -- 172.21.15.16:0/1222148286 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcf180f2480 con 0x7fcf20105df0 2024-04-13T03:31:57.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.722+0000 7fcf1de00640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf040a3bb0 0x7fcf040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.723+0000 7fcf1de00640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf040a3bb0 0x7fcf040a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcf20105290 tx=0x7fcf10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:57.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.727+0000 7fcf17400640 1 -- 172.21.15.16:0/1222148286 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcf1804b020 con 0x7fcf20105df0 2024-04-13T03:31:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.956+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcf20062bc0 con 0x7fcf20105df0 2024-04-13T03:31:57.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.960+0000 7fcf17400640 1 -- 172.21.15.16:0/1222148286 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+586 (secure 0 0 0) 0x7fcf180bec50 con 0x7fcf20105df0 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/164620940 2024-04-14T03:23:39.714975+0000 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/83197818 2024-04-14T03:23:04.480630+0000 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/20480070 2024-04-14T03:23:39.714975+0000 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3152772162 2024-04-14T03:23:04.480630+0000 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/2135228404 2024-04-14T03:23:04.480630+0000 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2468290259 2024-04-14T03:23:39.714975+0000 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/164620940 2024-04-14T03:23:39.714975+0000 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1664031642 2024-04-14T03:23:04.480630+0000 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3312707257 2024-04-14T03:23:39.714975+0000 2024-04-13T03:31:57.962 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/2135228404 2024-04-14T03:23:04.480630+0000 2024-04-13T03:31:57.963 INFO:teuthology.orchestra.run.smithi016.stderr:listed 10 entries 2024-04-13T03:31:57.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf040a3bb0 msgr2=0x7fcf040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf040a3bb0 0x7fcf040a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcf20105290 tx=0x7fcf10005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 msgr2=0x7fcf201a1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 0x7fcf201a1a70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcf1802c630 tx=0x7fcf18032ea0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 shutdown_connections 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf040a3bb0 0x7fcf040a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf20105df0 0x7fcf201a1a70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf20105430 0x7fcf2019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 --2- 172.21.15.16:0/1222148286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf20104240 0x7fcf2019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 >> 172.21.15.16:0/1222148286 conn(0x7fcf200ff7b0 msgr2=0x7fcf20108730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 shutdown_connections 2024-04-13T03:31:57.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:57.964+0000 7fcf25000640 1 -- 172.21.15.16:0/1222148286 wait complete. 2024-04-13T03:31:57.984 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:31:57.984 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-13T03:31:58.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:31:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.261+0000 7f1995200640 1 -- 172.21.15.16:0/3187773874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1990104040 msgr2=0x7f1990104460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.261+0000 7f1995200640 1 --2- 172.21.15.16:0/3187773874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1990104040 0x7f1990104460 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1980009900 tx=0x7f198002bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.262+0000 7f1995200640 1 -- 172.21.15.16:0/3187773874 shutdown_connections 2024-04-13T03:31:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.262+0000 7f1995200640 1 --2- 172.21.15.16:0/3187773874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1990105bf0 0x7f199010c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.262+0000 7f1995200640 1 --2- 172.21.15.16:0/3187773874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1990105230 0x7f19901056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.262+0000 7f1995200640 1 --2- 172.21.15.16:0/3187773874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1990104040 0x7f1990104460 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.262+0000 7f1995200640 1 -- 172.21.15.16:0/3187773874 >> 172.21.15.16:0/3187773874 conn(0x7f19900ff5d0 msgr2=0x7f19901019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.262+0000 7f1995200640 1 -- 172.21.15.16:0/3187773874 shutdown_connections 2024-04-13T03:31:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.262+0000 7f1995200640 1 -- 172.21.15.16:0/3187773874 wait complete. 2024-04-13T03:31:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.263+0000 7f1995200640 1 Processor -- start 2024-04-13T03:31:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.263+0000 7f1995200640 1 -- start start 2024-04-13T03:31:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.263+0000 7f1995200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1990104040 0x7f199019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.263+0000 7f1995200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1990105230 0x7f199019a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.263+0000 7f1995200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1990105bf0 0x7f19901a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f1995200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f199010fce0 con 0x7f1990105230 2024-04-13T03:31:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f1995200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f199010fb60 con 0x7f1990104040 2024-04-13T03:31:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f1995200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f199010fe60 con 0x7f1990105bf0 2024-04-13T03:31:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1990105230 0x7f199019a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1990105bf0 0x7f19901a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1990104040 0x7f199019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1990105230 0x7f199019a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56944/0 (socket says 172.21.15.16:56944) 2024-04-13T03:31:58.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198ea00640 1 -- 172.21.15.16:0/3944971949 learned_addr learned my addr 172.21.15.16:0/3944971949 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198fe00640 1 -- 172.21.15.16:0/3944971949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1990104040 msgr2=0x7f199019a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198fe00640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1990104040 0x7f199019a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198fe00640 1 -- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1990105230 msgr2=0x7f199019a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198fe00640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1990105230 0x7f199019a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198fe00640 1 -- 172.21.15.16:0/3944971949 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19901a1f40 con 0x7f1990105bf0 2024-04-13T03:31:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.264+0000 7f198ea00640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1990105230 0x7f199019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:31:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.265+0000 7f198fe00640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1990105bf0 0x7f19901a18d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1988009fd0 tx=0x7f198800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.265+0000 7f1987e00640 1 -- 172.21.15.16:0/3944971949 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198800ce20 con 0x7f1990105bf0 2024-04-13T03:31:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.265+0000 7f1987e00640 1 -- 172.21.15.16:0/3944971949 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1988005020 con 0x7f1990105bf0 2024-04-13T03:31:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.265+0000 7f1987e00640 1 -- 172.21.15.16:0/3944971949 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988010720 con 0x7f1990105bf0 2024-04-13T03:31:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.265+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19901a2230 con 0x7f1990105bf0 2024-04-13T03:31:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.265+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19901a2780 con 0x7f1990105bf0 2024-04-13T03:31:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.266+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f199010b360 con 0x7f1990105bf0 2024-04-13T03:31:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.268+0000 7f1987e00640 1 -- 172.21.15.16:0/3944971949 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1988005540 con 0x7f1990105bf0 2024-04-13T03:31:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.270+0000 7f1987e00640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19740a3c30 0x7f19740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.270+0000 7f1987e00640 1 -- 172.21.15.16:0/3944971949 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f19880d1150 con 0x7f1990105bf0 2024-04-13T03:31:58.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.270+0000 7f198f400640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19740a3c30 0x7f19740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.271+0000 7f198f400640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19740a3c30 0x7f19740a60f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f198002c170 tx=0x7f198002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.272+0000 7f1987e00640 1 -- 172.21.15.16:0/3944971949 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f198809d890 con 0x7f1990105bf0 2024-04-13T03:31:58.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:57 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1133464047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:31:58.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:57 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2475233279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:31:58.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:57 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1133464047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:31:58.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:57 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2475233279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:31:58.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:57 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1133464047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:31:58.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:57 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2475233279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:31:58.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.482+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1990062bc0 con 0x7f1990105bf0 2024-04-13T03:31:58.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.484+0000 7f1987e00640 1 -- 172.21.15.16:0/3944971949 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7f198809d190 con 0x7f1990105bf0 2024-04-13T03:31:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:31:58.486 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.487+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19740a3c30 msgr2=0x7f19740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.487+0000 7f1995200640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19740a3c30 0x7f19740a60f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f198002c170 tx=0x7f198002df90 comp rx=0 tx=0).stop 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.487+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1990105bf0 msgr2=0x7f19901a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.487+0000 7f1995200640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1990105bf0 0x7f19901a18d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1988009fd0 tx=0x7f198800ef10 comp rx=0 tx=0).stop 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.488+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 shutdown_connections 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.488+0000 7f1995200640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19740a3c30 0x7f19740a60f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.488+0000 7f1995200640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1990105bf0 0x7f19901a18d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.488+0000 7f1995200640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1990105230 0x7f199019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.488+0000 7f1995200640 1 --2- 172.21.15.16:0/3944971949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1990104040 0x7f199019a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.488+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 >> 172.21.15.16:0/3944971949 conn(0x7f19900ff5d0 msgr2=0x7f1990108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:58.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.488+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 shutdown_connections 2024-04-13T03:31:58.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.488+0000 7f1995200640 1 -- 172.21.15.16:0/3944971949 wait complete. 2024-04-13T03:31:58.508 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-13T03:31:34.802688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-13T03:31:36.036786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-13T03:31:39.824413+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-13T03:31:58.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:31:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.734+0000 7fa00e000640 1 -- 172.21.15.16:0/2568663530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa008108540 msgr2=0x7fa008108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.734+0000 7fa00e000640 1 --2- 172.21.15.16:0/2568663530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa008108540 0x7fa008108960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009900 tx=0x7f9ffc02bc40 comp rx=0 tx=0).stop 2024-04-13T03:31:58.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.734+0000 7fa00e000640 1 -- 172.21.15.16:0/2568663530 shutdown_connections 2024-04-13T03:31:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.734+0000 7fa00e000640 1 --2- 172.21.15.16:0/2568663530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810a0f0 0x7fa008110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.734+0000 7fa00e000640 1 --2- 172.21.15.16:0/2568663530 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa008109730 0x7fa008109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.734+0000 7fa00e000640 1 --2- 172.21.15.16:0/2568663530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa008108540 0x7fa008108960 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.734+0000 7fa00e000640 1 -- 172.21.15.16:0/2568663530 >> 172.21.15.16:0/2568663530 conn(0x7fa0080774f0 msgr2=0x7fa008079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.734+0000 7fa00e000640 1 -- 172.21.15.16:0/2568663530 shutdown_connections 2024-04-13T03:31:58.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.734+0000 7fa00e000640 1 -- 172.21.15.16:0/2568663530 wait complete. 2024-04-13T03:31:58.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.735+0000 7fa00e000640 1 Processor -- start 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.735+0000 7fa00e000640 1 -- start start 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00e000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa008108540 0x7fa00819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa008109730 0x7fa00819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810a0f0 0x7fa0081a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00e000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0081141a0 con 0x7fa008108540 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00e000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa008114020 con 0x7fa008109730 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00e000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa008114320 con 0x7fa00810a0f0 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa008108540 0x7fa00819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa008108540 0x7fa00819e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56968/0 (socket says 172.21.15.16:56968) 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00cc00640 1 -- 172.21.15.16:0/1947862473 learned_addr learned my addr 172.21.15.16:0/1947862473 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa007e00640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa008109730 0x7fa00819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.736+0000 7fa00d600640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810a0f0 0x7fa0081a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:58.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa007e00640 1 -- 172.21.15.16:0/1947862473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810a0f0 msgr2=0x7fa0081a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa007e00640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810a0f0 0x7fa0081a5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa007e00640 1 -- 172.21.15.16:0/1947862473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa008108540 msgr2=0x7fa00819e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa007e00640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa008108540 0x7fa00819e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa007e00640 1 -- 172.21.15.16:0/1947862473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0081a64a0 con 0x7fa008109730 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa007e00640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa008109730 0x7fa00819ee10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400e9f0 tx=0x7f9ff400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa005600640 1 -- 172.21.15.16:0/1947862473 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff400ccc0 con 0x7fa008109730 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa005600640 1 -- 172.21.15.16:0/1947862473 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ff4010910 con 0x7fa008109730 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa005600640 1 -- 172.21.15.16:0/1947862473 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4018960 con 0x7fa008109730 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.737+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0081a6790 con 0x7fa008109730 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.738+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0081a6c90 con 0x7fa008109730 2024-04-13T03:31:58.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.739+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa00810f8e0 con 0x7fa008109730 2024-04-13T03:31:58.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.741+0000 7fa005600640 1 -- 172.21.15.16:0/1947862473 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9ff4010ac0 con 0x7fa008109730 2024-04-13T03:31:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.743+0000 7fa005600640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9ff00a3bb0 0x7f9ff00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.743+0000 7fa005600640 1 -- 172.21.15.16:0/1947862473 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9ff40d0a90 con 0x7fa008109730 2024-04-13T03:31:58.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.743+0000 7fa00cc00640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9ff00a3bb0 0x7f9ff00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.744+0000 7fa00cc00640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9ff00a3bb0 0x7f9ff00a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc02c8f0 tx=0x7f9ffc005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:58.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.747+0000 7fa005600640 1 -- 172.21.15.16:0/1947862473 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ff409d1d0 con 0x7fa008109730 2024-04-13T03:31:58.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.953+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa008062bc0 con 0x7fa008109730 2024-04-13T03:31:58.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.954+0000 7fa005600640 1 -- 172.21.15.16:0/1947862473 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f9ff409cad0 con 0x7fa008109730 2024-04-13T03:31:58.958 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9ff00a3bb0 msgr2=0x7f9ff00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9ff00a3bb0 0x7f9ff00a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc02c8f0 tx=0x7f9ffc005b50 comp rx=0 tx=0).stop 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa008109730 msgr2=0x7fa00819ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa008109730 0x7fa00819ee10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400e9f0 tx=0x7f9ff400eeb0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 shutdown_connections 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9ff00a3bb0 0x7f9ff00a6070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa00810a0f0 0x7fa0081a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa008109730 0x7fa00819ee10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 --2- 172.21.15.16:0/1947862473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa008108540 0x7fa00819e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:58.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 >> 172.21.15.16:0/1947862473 conn(0x7fa0080774f0 msgr2=0x7fa00810cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:58.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.956+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 shutdown_connections 2024-04-13T03:31:58.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:58.957+0000 7fa00e000640 1 -- 172.21.15.16:0/1947862473 wait complete. 2024-04-13T03:31:58.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-13T03:31:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.194+0000 7f3f5c000640 1 -- 172.21.15.16:0/1939218146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f54108620 msgr2=0x7f3f54108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.194+0000 7f3f5c000640 1 --2- 172.21.15.16:0/1939218146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f54108620 0x7f3f54108a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009900 tx=0x7f3f4802bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.194+0000 7f3f5c000640 1 -- 172.21.15.16:0/1939218146 shutdown_connections 2024-04-13T03:31:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.194+0000 7f3f5c000640 1 --2- 172.21.15.16:0/1939218146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f5410a1d0 0x7f3f54110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.194+0000 7f3f5c000640 1 --2- 172.21.15.16:0/1939218146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f54109810 0x7f3f54109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.194+0000 7f3f5c000640 1 --2- 172.21.15.16:0/1939218146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f54108620 0x7f3f54108a40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.194+0000 7f3f5c000640 1 -- 172.21.15.16:0/1939218146 >> 172.21.15.16:0/1939218146 conn(0x7f3f540775c0 msgr2=0x7f3f54079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.195+0000 7f3f5c000640 1 -- 172.21.15.16:0/1939218146 shutdown_connections 2024-04-13T03:31:59.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.195+0000 7f3f5c000640 1 -- 172.21.15.16:0/1939218146 wait complete. 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.196+0000 7f3f5c000640 1 Processor -- start 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.196+0000 7f3f5c000640 1 -- start start 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.196+0000 7f3f5c000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f54108620 0x7f3f5419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.196+0000 7f3f5c000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f54109810 0x7f3f5419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.196+0000 7f3f5c000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f5410a1d0 0x7f3f541a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.196+0000 7f3f5c000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541142a0 con 0x7f3f54109810 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.196+0000 7f3f5c000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54114120 con 0x7f3f5410a1d0 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.196+0000 7f3f5c000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54114420 con 0x7f3f54108620 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f59600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f54108620 0x7f3f5419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f59600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f54108620 0x7f3f5419e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:49218/0 (socket says 172.21.15.16:49218) 2024-04-13T03:31:59.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f59600640 1 -- 172.21.15.16:0/3657436184 learned_addr learned my addr 172.21.15.16:0/3657436184 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:59.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f58c00640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f54109810 0x7f3f5419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:59.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f5a000640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f5410a1d0 0x7f3f541a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f59600640 1 -- 172.21.15.16:0/3657436184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f5410a1d0 msgr2=0x7f3f541a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f59600640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f5410a1d0 0x7f3f541a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f59600640 1 -- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f54109810 msgr2=0x7f3f5419eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f59600640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f54109810 0x7f3f5419eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.197+0000 7f3f59600640 1 -- 172.21.15.16:0/3657436184 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f541a6590 con 0x7f3f54108620 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.198+0000 7f3f59600640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f54108620 0x7f3f5419e970 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3f48000c00 tx=0x7f3f4802c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.198+0000 7f3f52000640 1 -- 172.21.15.16:0/3657436184 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48041070 con 0x7f3f54108620 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.198+0000 7f3f52000640 1 -- 172.21.15.16:0/3657436184 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f48004450 con 0x7f3f54108620 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.198+0000 7f3f52000640 1 -- 172.21.15.16:0/3657436184 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48034440 con 0x7f3f54108620 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.198+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f541a6820 con 0x7f3f54108620 2024-04-13T03:31:59.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.198+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f541a6cc0 con 0x7f3f54108620 2024-04-13T03:31:59.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.199+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f5410fa10 con 0x7f3f54108620 2024-04-13T03:31:59.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.201+0000 7f3f52000640 1 -- 172.21.15.16:0/3657436184 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3f48044070 con 0x7f3f54108620 2024-04-13T03:31:59.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.203+0000 7f3f52000640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.203+0000 7f3f52000640 1 -- 172.21.15.16:0/3657436184 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3f480f2180 con 0x7f3f54108620 2024-04-13T03:31:59.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.203+0000 7f3f58c00640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:59.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.204+0000 7f3f58c00640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3f5419fe80 tx=0x7f3f40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:59.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.207+0000 7f3f52000640 1 -- 172.21.15.16:0/3657436184 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f480be300 con 0x7f3f54108620 2024-04-13T03:31:59.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:58 smithi119 ceph-mon[35835]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-13T03:31:59.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:58 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1222148286' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-13T03:31:59.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:58 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3944971949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:59.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:58 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1947862473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:31:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:58 smithi016 ceph-mon[31772]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-13T03:31:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:58 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1222148286' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-13T03:31:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:58 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3944971949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:59.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:58 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1947862473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:31:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:58 smithi188 ceph-mon[36398]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-13T03:31:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:58 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1222148286' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-13T03:31:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:58 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3944971949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:31:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:58 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1947862473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:31:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.412+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3f54062bf0 con 0x7f3f54108620 2024-04-13T03:31:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.413+0000 7f3f52000640 1 -- 172.21.15.16:0/3657436184 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f3f480c2600 con 0x7f3f54108620 2024-04-13T03:31:59.414 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.415+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f3c0a3bb0 msgr2=0x7f3f3c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.415+0000 7f3f5c000640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3f5419fe80 tx=0x7f3f40005f90 comp rx=0 tx=0).stop 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.415+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f54108620 msgr2=0x7f3f5419e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.415+0000 7f3f5c000640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f54108620 0x7f3f5419e970 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3f48000c00 tx=0x7f3f4802c640 comp rx=0 tx=0).stop 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.415+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 shutdown_connections 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.415+0000 7f3f5c000640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3f3c0a3bb0 0x7f3f3c0a6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.416+0000 7f3f5c000640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f5410a1d0 0x7f3f541a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.416+0000 7f3f5c000640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f54109810 0x7f3f5419eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.416+0000 7f3f5c000640 1 --2- 172.21.15.16:0/3657436184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f54108620 0x7f3f5419e970 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.416+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 >> 172.21.15.16:0/3657436184 conn(0x7f3f540775c0 msgr2=0x7f3f5410cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.416+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 shutdown_connections 2024-04-13T03:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.416+0000 7f3f5c000640 1 -- 172.21.15.16:0/3657436184 wait complete. 2024-04-13T03:31:59.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-13T03:31:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.672+0000 7f174fe00640 1 -- 172.21.15.16:0/2878144721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17501052f0 msgr2=0x7f1750105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.672+0000 7f174fe00640 1 --2- 172.21.15.16:0/2878144721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17501052f0 0x7f1750105770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1738009900 tx=0x7f173802bc30 comp rx=0 tx=0).stop 2024-04-13T03:31:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.672+0000 7f174fe00640 1 -- 172.21.15.16:0/2878144721 shutdown_connections 2024-04-13T03:31:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.672+0000 7f174fe00640 1 --2- 172.21.15.16:0/2878144721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1750105cb0 0x7f175010c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.672+0000 7f174fe00640 1 --2- 172.21.15.16:0/2878144721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17501052f0 0x7f1750105770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.672+0000 7f174fe00640 1 --2- 172.21.15.16:0/2878144721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750104100 0x7f1750104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.672+0000 7f174fe00640 1 -- 172.21.15.16:0/2878144721 >> 172.21.15.16:0/2878144721 conn(0x7f17500ff670 msgr2=0x7f1750101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:59.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.672+0000 7f174fe00640 1 -- 172.21.15.16:0/2878144721 shutdown_connections 2024-04-13T03:31:59.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.672+0000 7f174fe00640 1 -- 172.21.15.16:0/2878144721 wait complete. 2024-04-13T03:31:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.673+0000 7f174fe00640 1 Processor -- start 2024-04-13T03:31:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.673+0000 7f174fe00640 1 -- start start 2024-04-13T03:31:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.673+0000 7f174fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750104100 0x7f175019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.673+0000 7f174fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17501052f0 0x7f175019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.673+0000 7f174fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1750105cb0 0x7f17501a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.673+0000 7f174fe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f175010fde0 con 0x7f1750104100 2024-04-13T03:31:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.673+0000 7f174fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f175010fc60 con 0x7f17501052f0 2024-04-13T03:31:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.673+0000 7f174fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f175010ff60 con 0x7f1750105cb0 2024-04-13T03:31:59.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750104100 0x7f175019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750104100 0x7f175019a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36746/0 (socket says 172.21.15.16:36746) 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174ea00640 1 -- 172.21.15.16:0/312359796 learned_addr learned my addr 172.21.15.16:0/312359796 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1750105cb0 0x7f17501a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174e000640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17501052f0 0x7f175019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174ea00640 1 -- 172.21.15.16:0/312359796 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1750105cb0 msgr2=0x7f17501a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174ea00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1750105cb0 0x7f17501a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174ea00640 1 -- 172.21.15.16:0/312359796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17501052f0 msgr2=0x7f175019a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174ea00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17501052f0 0x7f175019a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174ea00640 1 -- 172.21.15.16:0/312359796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17501a1fd0 con 0x7f1750104100 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.674+0000 7f174ea00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750104100 0x7f175019a3b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f174000e9c0 tx=0x7f174000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.675+0000 7f1754a00640 1 -- 172.21.15.16:0/312359796 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174000cd80 con 0x7f1750104100 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.675+0000 7f1754a00640 1 -- 172.21.15.16:0/312359796 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17400035f0 con 0x7f1750104100 2024-04-13T03:31:59.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.675+0000 7f1754a00640 1 -- 172.21.15.16:0/312359796 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740010690 con 0x7f1750104100 2024-04-13T03:31:59.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.675+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17501a22c0 con 0x7f1750104100 2024-04-13T03:31:59.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.675+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17501a27e0 con 0x7f1750104100 2024-04-13T03:31:59.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.676+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f175010b420 con 0x7f1750104100 2024-04-13T03:31:59.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.678+0000 7f1754a00640 1 -- 172.21.15.16:0/312359796 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f17400037a0 con 0x7f1750104100 2024-04-13T03:31:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.679+0000 7f1754a00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f17340a3bb0 0x7f17340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.680+0000 7f1754a00640 1 -- 172.21.15.16:0/312359796 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f17400d0aa0 con 0x7f1750104100 2024-04-13T03:31:59.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.680+0000 7f174e000640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f17340a3bb0 0x7f17340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:59.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.681+0000 7f174e000640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f17340a3bb0 0x7f17340a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f173802c240 tx=0x7f173802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.685+0000 7f1754a00640 1 -- 172.21.15.16:0/312359796 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f174009d1e0 con 0x7f1750104100 2024-04-13T03:31:59.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.894+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1750062bc0 con 0x7f1750104100 2024-04-13T03:31:59.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.895+0000 7f1754a00640 1 -- 172.21.15.16:0/312359796 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f174009cae0 con 0x7f1750104100 2024-04-13T03:31:59.896 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-13T03:31:59.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f17340a3bb0 msgr2=0x7f17340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f17340a3bb0 0x7f17340a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f173802c240 tx=0x7f173802df90 comp rx=0 tx=0).stop 2024-04-13T03:31:59.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750104100 msgr2=0x7f175019a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:31:59.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750104100 0x7f175019a3b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f174000e9c0 tx=0x7f174000ee80 comp rx=0 tx=0).stop 2024-04-13T03:31:59.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 shutdown_connections 2024-04-13T03:31:59.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f17340a3bb0 0x7f17340a6070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1750105cb0 0x7f17501a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f17501052f0 0x7f175019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 --2- 172.21.15.16:0/312359796 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1750104100 0x7f175019a3b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:59.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 >> 172.21.15.16:0/312359796 conn(0x7f17500ff670 msgr2=0x7f17501085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:31:59.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 shutdown_connections 2024-04-13T03:31:59.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:31:59.898+0000 7f174fe00640 1 -- 172.21.15.16:0/312359796 wait complete. 2024-04-13T03:31:59.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-13T03:32:00.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.139+0000 7fb6bf200640 1 -- 172.21.15.16:0/3857812982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6c01052f0 msgr2=0x7fb6c0105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.139+0000 7fb6bf200640 1 --2- 172.21.15.16:0/3857812982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6c01052f0 0x7fb6c0105770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009900 tx=0x7fb6ac02bc50 comp rx=0 tx=0).stop 2024-04-13T03:32:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.139+0000 7fb6bf200640 1 -- 172.21.15.16:0/3857812982 shutdown_connections 2024-04-13T03:32:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.139+0000 7fb6bf200640 1 --2- 172.21.15.16:0/3857812982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c0105cb0 0x7fb6c010c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.139+0000 7fb6bf200640 1 --2- 172.21.15.16:0/3857812982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6c01052f0 0x7fb6c0105770 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.139+0000 7fb6bf200640 1 --2- 172.21.15.16:0/3857812982 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c0104100 0x7fb6c0104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.139+0000 7fb6bf200640 1 -- 172.21.15.16:0/3857812982 >> 172.21.15.16:0/3857812982 conn(0x7fb6c00ff670 msgr2=0x7fb6c0101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.139+0000 7fb6bf200640 1 -- 172.21.15.16:0/3857812982 shutdown_connections 2024-04-13T03:32:00.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.139+0000 7fb6bf200640 1 -- 172.21.15.16:0/3857812982 wait complete. 2024-04-13T03:32:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.140+0000 7fb6bf200640 1 Processor -- start 2024-04-13T03:32:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.140+0000 7fb6bf200640 1 -- start start 2024-04-13T03:32:00.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.140+0000 7fb6bf200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c0104100 0x7fb6c019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.140+0000 7fb6bf200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c01052f0 0x7fb6c019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.140+0000 7fb6bf200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6c0105cb0 0x7fb6c01a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.140+0000 7fb6bf200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c010fde0 con 0x7fb6c01052f0 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.140+0000 7fb6bf200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c010fc60 con 0x7fb6c0105cb0 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.140+0000 7fb6bf200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c010ff60 con 0x7fb6c0104100 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6bd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c01052f0 0x7fb6c019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6bd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c01052f0 0x7fb6c019a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36764/0 (socket says 172.21.15.16:36764) 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6bd400640 1 -- 172.21.15.16:0/450693757 learned_addr learned my addr 172.21.15.16:0/450693757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6bde00640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c0104100 0x7fb6c019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6be800640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6c0105cb0 0x7fb6c01a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6bd400640 1 -- 172.21.15.16:0/450693757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c0104100 msgr2=0x7fb6c019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6bd400640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c0104100 0x7fb6c019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6bd400640 1 -- 172.21.15.16:0/450693757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6c0105cb0 msgr2=0x7fb6c01a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6bd400640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6c0105cb0 0x7fb6c01a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.141+0000 7fb6bd400640 1 -- 172.21.15.16:0/450693757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c01a2030 con 0x7fb6c01052f0 2024-04-13T03:32:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.142+0000 7fb6bd400640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c01052f0 0x7fb6c019a950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac005ab0 tx=0x7fb6ac02c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.142+0000 7fb6b6a00640 1 -- 172.21.15.16:0/450693757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac041070 con 0x7fb6c01052f0 2024-04-13T03:32:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.142+0000 7fb6b6a00640 1 -- 172.21.15.16:0/450693757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6ac004450 con 0x7fb6c01052f0 2024-04-13T03:32:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.142+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c01a22c0 con 0x7fb6c01052f0 2024-04-13T03:32:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.142+0000 7fb6b6a00640 1 -- 172.21.15.16:0/450693757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac034440 con 0x7fb6c01052f0 2024-04-13T03:32:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.142+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c01a2810 con 0x7fb6c01052f0 2024-04-13T03:32:00.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.143+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c010b420 con 0x7fb6c01052f0 2024-04-13T03:32:00.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.146+0000 7fb6b6a00640 1 -- 172.21.15.16:0/450693757 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb6ac044070 con 0x7fb6c01052f0 2024-04-13T03:32:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.147+0000 7fb6b6a00640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6a40a3bb0 0x7fb6a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.147+0000 7fb6b6a00640 1 -- 172.21.15.16:0/450693757 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb6ac0f25e0 con 0x7fb6c01052f0 2024-04-13T03:32:00.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.147+0000 7fb6bde00640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6a40a3bb0 0x7fb6a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:00.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.148+0000 7fb6bde00640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6a40a3bb0 0x7fb6a40a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0105150 tx=0x7fb6b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:00.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.150+0000 7fb6b6a00640 1 -- 172.21.15.16:0/450693757 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6ac0be6d0 con 0x7fb6c01052f0 2024-04-13T03:32:00.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:59 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3657436184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:32:00.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:31:59 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/312359796' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:32:00.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:59 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3657436184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:32:00.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:31:59 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/312359796' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:32:00.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.355+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb6c002d830 con 0x7fb6c01052f0 2024-04-13T03:32:00.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.356+0000 7fb6b6a00640 1 -- 172.21.15.16:0/450693757 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fb6ac0c29d0 con 0x7fb6c01052f0 2024-04-13T03:32:00.357 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.358+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6a40a3bb0 msgr2=0x7fb6a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.358+0000 7fb6bf200640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6a40a3bb0 0x7fb6a40a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0105150 tx=0x7fb6b0005f90 comp rx=0 tx=0).stop 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c01052f0 msgr2=0x7fb6c019a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c01052f0 0x7fb6c019a950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac005ab0 tx=0x7fb6ac02c6e0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 shutdown_connections 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6a40a3bb0 0x7fb6a40a6070 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6c0105cb0 0x7fb6c01a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6c01052f0 0x7fb6c019a950 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 --2- 172.21.15.16:0/450693757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6c0104100 0x7fb6c019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 >> 172.21.15.16:0/450693757 conn(0x7fb6c00ff670 msgr2=0x7fb6c01085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 shutdown_connections 2024-04-13T03:32:00.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.359+0000 7fb6bf200640 1 -- 172.21.15.16:0/450693757 wait complete. 2024-04-13T03:32:00.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:59 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3657436184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:32:00.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:31:59 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/312359796' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:32:00.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-13T03:32:00.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.609+0000 7fb6d1a00640 1 -- 172.21.15.16:0/1739376363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6cc104100 msgr2=0x7fb6cc104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.609+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/1739376363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6cc104100 0x7fb6cc104520 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009860 tx=0x7fb6bc02bb70 comp rx=0 tx=0).stop 2024-04-13T03:32:00.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.610+0000 7fb6d1a00640 1 -- 172.21.15.16:0/1739376363 shutdown_connections 2024-04-13T03:32:00.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.610+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/1739376363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6cc105cb0 0x7fb6cc10c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.610+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/1739376363 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6cc1052f0 0x7fb6cc105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.610+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/1739376363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6cc104100 0x7fb6cc104520 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.610+0000 7fb6d1a00640 1 -- 172.21.15.16:0/1739376363 >> 172.21.15.16:0/1739376363 conn(0x7fb6cc0ff670 msgr2=0x7fb6cc101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:00.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.610+0000 7fb6d1a00640 1 -- 172.21.15.16:0/1739376363 shutdown_connections 2024-04-13T03:32:00.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.610+0000 7fb6d1a00640 1 -- 172.21.15.16:0/1739376363 wait complete. 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6d1a00640 1 Processor -- start 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6d1a00640 1 -- start start 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6d1a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6cc104100 0x7fb6cc19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6d1a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6cc1052f0 0x7fb6cc19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6d1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6cc105cb0 0x7fb6cc1a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6d1a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6cc10fd90 con 0x7fb6cc105cb0 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6d1a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb6cc10fc10 con 0x7fb6cc1052f0 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6d1a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb6cc10ff10 con 0x7fb6cc104100 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6d1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6cc105cb0 0x7fb6cc1a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6cbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6cc104100 0x7fb6cc19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6cbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6cc104100 0x7fb6cc19a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:49270/0 (socket says 172.21.15.16:49270) 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.611+0000 7fb6cbe00640 1 -- 172.21.15.16:0/3445778309 learned_addr learned my addr 172.21.15.16:0/3445778309 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:00.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.612+0000 7fb6c3400640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6cc1052f0 0x7fb6cc19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.612+0000 7fb6cbe00640 1 -- 172.21.15.16:0/3445778309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6cc1052f0 msgr2=0x7fb6cc19a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.612+0000 7fb6cbe00640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6cc1052f0 0x7fb6cc19a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.612+0000 7fb6cbe00640 1 -- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6cc105cb0 msgr2=0x7fb6cc1a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.612+0000 7fb6cbe00640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6cc105cb0 0x7fb6cc1a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.612+0000 7fb6cbe00640 1 -- 172.21.15.16:0/3445778309 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6cc1a2030 con 0x7fb6cc104100 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.612+0000 7fb6d1000640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6cc105cb0 0x7fb6cc1a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.612+0000 7fb6cbe00640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6cc104100 0x7fb6cc19a410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0349e0 tx=0x7fb6bc032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.612+0000 7fb6c9600640 1 -- 172.21.15.16:0/3445778309 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc041020 con 0x7fb6cc104100 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.613+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6cc1a22c0 con 0x7fb6cc104100 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.613+0000 7fb6c9600640 1 -- 172.21.15.16:0/3445778309 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6bc02c9f0 con 0x7fb6cc104100 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.613+0000 7fb6c9600640 1 -- 172.21.15.16:0/3445778309 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc039040 con 0x7fb6cc104100 2024-04-13T03:32:00.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.613+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6cc1a2800 con 0x7fb6cc104100 2024-04-13T03:32:00.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.614+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6cc10b420 con 0x7fb6cc104100 2024-04-13T03:32:00.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.616+0000 7fb6c9600640 1 -- 172.21.15.16:0/3445778309 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb6bc044020 con 0x7fb6cc104100 2024-04-13T03:32:00.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.617+0000 7fb6c9600640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6b00a3bb0 0x7fb6b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:00.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.617+0000 7fb6c9600640 1 -- 172.21.15.16:0/3445778309 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb6bc0f2290 con 0x7fb6cc104100 2024-04-13T03:32:00.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.617+0000 7fb6c3400640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6b00a3bb0 0x7fb6b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:00.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.618+0000 7fb6c3400640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6b00a3bb0 0x7fb6b00a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc19b920 tx=0x7fb6b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:00.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.619+0000 7fb6c9600640 1 -- 172.21.15.16:0/3445778309 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6bc04b020 con 0x7fb6cc104100 2024-04-13T03:32:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.842+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb6cc02d5e0 con 0x7fb6cc104100 2024-04-13T03:32:00.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.843+0000 7fb6c9600640 1 -- 172.21.15.16:0/3445778309 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+586 (secure 0 0 0) 0x7fb6bc0be9d0 con 0x7fb6cc104100 2024-04-13T03:32:00.844 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/164620940 2024-04-14T03:23:39.714975+0000 2024-04-13T03:32:00.844 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/83197818 2024-04-14T03:23:04.480630+0000 2024-04-13T03:32:00.844 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/20480070 2024-04-14T03:23:39.714975+0000 2024-04-13T03:32:00.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3152772162 2024-04-14T03:23:04.480630+0000 2024-04-13T03:32:00.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/2135228404 2024-04-14T03:23:04.480630+0000 2024-04-13T03:32:00.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2468290259 2024-04-14T03:23:39.714975+0000 2024-04-13T03:32:00.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/164620940 2024-04-14T03:23:39.714975+0000 2024-04-13T03:32:00.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1664031642 2024-04-14T03:23:04.480630+0000 2024-04-13T03:32:00.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3312707257 2024-04-14T03:23:39.714975+0000 2024-04-13T03:32:00.845 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/2135228404 2024-04-14T03:23:04.480630+0000 2024-04-13T03:32:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:listed 10 entries 2024-04-13T03:32:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.845+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6b00a3bb0 msgr2=0x7fb6b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.845+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6b00a3bb0 0x7fb6b00a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc19b920 tx=0x7fb6b4009260 comp rx=0 tx=0).stop 2024-04-13T03:32:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6cc104100 msgr2=0x7fb6cc19a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6cc104100 0x7fb6cc19a410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0349e0 tx=0x7fb6bc032430 comp rx=0 tx=0).stop 2024-04-13T03:32:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 shutdown_connections 2024-04-13T03:32:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6b00a3bb0 0x7fb6b00a6070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6cc105cb0 0x7fb6cc1a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6cc1052f0 0x7fb6cc19a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 --2- 172.21.15.16:0/3445778309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6cc104100 0x7fb6cc19a410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 >> 172.21.15.16:0/3445778309 conn(0x7fb6cc0ff670 msgr2=0x7fb6cc1085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 shutdown_connections 2024-04-13T03:32:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:00.846+0000 7fb6d1a00640 1 -- 172.21.15.16:0/3445778309 wait complete. 2024-04-13T03:32:00.867 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-04-13T03:32:00.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:32:00.867 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-13T03:32:00.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[72692]: fuse finished with error 0 and tester_r 0 2024-04-13T03:32:00.974 INFO:teuthology.orchestra.run:waiting for 300 2024-04-13T03:32:01.040 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T03:32:01.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:32:01.040 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-13T03:32:01.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:32:01.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:00 smithi016 ceph-mon[31772]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-13T03:32:01.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:00 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/450693757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:32:01.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:00 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3445778309' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-13T03:32:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:00 smithi119 ceph-mon[35835]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-13T03:32:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:00 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/450693757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:32:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:00 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3445778309' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-13T03:32:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.349+0000 7f672e200640 1 -- 172.21.15.16:0/1529520308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6728109630 msgr2=0x7f6728109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.349+0000 7f672e200640 1 --2- 172.21.15.16:0/1529520308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6728109630 0x7f6728109ab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6710009860 tx=0x7f671002bb70 comp rx=0 tx=0).stop 2024-04-13T03:32:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.350+0000 7f672e200640 1 -- 172.21.15.16:0/1529520308 shutdown_connections 2024-04-13T03:32:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.350+0000 7f672e200640 1 --2- 172.21.15.16:0/1529520308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6728109ff0 0x7f6728110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.350+0000 7f672e200640 1 --2- 172.21.15.16:0/1529520308 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6728109630 0x7f6728109ab0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.350+0000 7f672e200640 1 --2- 172.21.15.16:0/1529520308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6728108440 0x7f6728108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.350+0000 7f672e200640 1 -- 172.21.15.16:0/1529520308 >> 172.21.15.16:0/1529520308 conn(0x7f6728077400 msgr2=0x7f6728079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.350+0000 7f672e200640 1 -- 172.21.15.16:0/1529520308 shutdown_connections 2024-04-13T03:32:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.350+0000 7f672e200640 1 -- 172.21.15.16:0/1529520308 wait complete. 2024-04-13T03:32:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.351+0000 7f672e200640 1 Processor -- start 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.351+0000 7f672e200640 1 -- start start 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.351+0000 7f672e200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6728108440 0x7f672819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.351+0000 7f672e200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6728109630 0x7f672819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.351+0000 7f672e200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6728109ff0 0x7f67281a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.351+0000 7f672e200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f67281140d0 con 0x7f6728108440 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.351+0000 7f672e200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6728113f50 con 0x7f6728109ff0 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.351+0000 7f672e200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6728114250 con 0x7f6728109630 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f672ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6728108440 0x7f672819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f672ce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6728108440 0x7f672819e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36816/0 (socket says 172.21.15.16:36816) 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f672ce00640 1 -- 172.21.15.16:0/1709625965 learned_addr learned my addr 172.21.15.16:0/1709625965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f671fe00640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6728109630 0x7f672819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f672d800640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6728109ff0 0x7f67281a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f672ce00640 1 -- 172.21.15.16:0/1709625965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6728109630 msgr2=0x7f672819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f672ce00640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6728109630 0x7f672819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f672ce00640 1 -- 172.21.15.16:0/1709625965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6728109ff0 msgr2=0x7f67281a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f672ce00640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6728109ff0 0x7f67281a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f672ce00640 1 -- 172.21.15.16:0/1709625965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67281a6320 con 0x7f6728108440 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.352+0000 7f671fe00640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6728109630 0x7f672819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.353+0000 7f672ce00640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6728108440 0x7f672819e750 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f671800e9a0 tx=0x7f671800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.353+0000 7f6726000640 1 -- 172.21.15.16:0/1709625965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671800cd80 con 0x7f6728108440 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.353+0000 7f6726000640 1 -- 172.21.15.16:0/1709625965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6718010910 con 0x7f6728108440 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.353+0000 7f6726000640 1 -- 172.21.15.16:0/1709625965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718018a30 con 0x7f6728108440 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.353+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67281a6610 con 0x7f6728108440 2024-04-13T03:32:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.353+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67281a6b80 con 0x7f6728108440 2024-04-13T03:32:01.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.354+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f672810f760 con 0x7f6728108440 2024-04-13T03:32:01.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.357+0000 7f6726000640 1 -- 172.21.15.16:0/1709625965 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6718010ac0 con 0x7f6728108440 2024-04-13T03:32:01.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.359+0000 7f6726000640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f670c0a3c30 0x7f670c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:01.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.359+0000 7f6726000640 1 -- 172.21.15.16:0/1709625965 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f67180d09a0 con 0x7f6728108440 2024-04-13T03:32:01.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.359+0000 7f671fe00640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f670c0a3c30 0x7f670c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:01.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.360+0000 7f671fe00640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f670c0a3c30 0x7f670c0a60f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6710009830 tx=0x7f671002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:01.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.363+0000 7f6726000640 1 -- 172.21.15.16:0/1709625965 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6718053ff0 con 0x7f6728108440 2024-04-13T03:32:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:00 smithi188 ceph-mon[36398]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-13T03:32:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:00 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/450693757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:32:01.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:00 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3445778309' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-13T03:32:01.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.569+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6728062bc0 con 0x7f6728108440 2024-04-13T03:32:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.571+0000 7f6726000640 1 -- 172.21.15.16:0/1709625965 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7f671809d0e0 con 0x7f6728108440 2024-04-13T03:32:01.573 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:32:01.573 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:32:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.574+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f670c0a3c30 msgr2=0x7f670c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.574+0000 7f672e200640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f670c0a3c30 0x7f670c0a60f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6710009830 tx=0x7f671002e040 comp rx=0 tx=0).stop 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.574+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6728108440 msgr2=0x7f672819e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.574+0000 7f672e200640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6728108440 0x7f672819e750 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f671800e9a0 tx=0x7f671800ee60 comp rx=0 tx=0).stop 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.574+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 shutdown_connections 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.574+0000 7f672e200640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f670c0a3c30 0x7f670c0a60f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.575+0000 7f672e200640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6728109ff0 0x7f67281a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.575+0000 7f672e200640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6728109630 0x7f672819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.575+0000 7f672e200640 1 --2- 172.21.15.16:0/1709625965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6728108440 0x7f672819e750 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.575+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 >> 172.21.15.16:0/1709625965 conn(0x7f6728077400 msgr2=0x7f672810c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.575+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 shutdown_connections 2024-04-13T03:32:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.575+0000 7f672e200640 1 -- 172.21.15.16:0/1709625965 wait complete. 2024-04-13T03:32:01.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:32:01.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.820+0000 7f46b3e00640 1 -- 172.21.15.16:0/1416896175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b4077bc0 msgr2=0x7f46b4078020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.820+0000 7f46b3e00640 1 --2- 172.21.15.16:0/1416896175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b4077bc0 0x7f46b4078020 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f469c009900 tx=0x7f469c02bc30 comp rx=0 tx=0).stop 2024-04-13T03:32:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.820+0000 7f46b3e00640 1 -- 172.21.15.16:0/1416896175 shutdown_connections 2024-04-13T03:32:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.820+0000 7f46b3e00640 1 --2- 172.21.15.16:0/1416896175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b4078560 0x7f46b4078e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.820+0000 7f46b3e00640 1 --2- 172.21.15.16:0/1416896175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b4077bc0 0x7f46b4078020 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.820+0000 7f46b3e00640 1 --2- 172.21.15.16:0/1416896175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46b4076980 0x7f46b4076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.820+0000 7f46b3e00640 1 -- 172.21.15.16:0/1416896175 >> 172.21.15.16:0/1416896175 conn(0x7f46b40ff5d0 msgr2=0x7f46b41019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.820+0000 7f46b3e00640 1 -- 172.21.15.16:0/1416896175 shutdown_connections 2024-04-13T03:32:01.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.820+0000 7f46b3e00640 1 -- 172.21.15.16:0/1416896175 wait complete. 2024-04-13T03:32:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.821+0000 7f46b3e00640 1 Processor -- start 2024-04-13T03:32:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.821+0000 7f46b3e00640 1 -- start start 2024-04-13T03:32:01.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46b4076980 0x7f46b419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b4077bc0 0x7f46b419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b4078560 0x7f46b41a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f46b41140d0 con 0x7f46b4078560 2024-04-13T03:32:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f46b4113f50 con 0x7f46b4076980 2024-04-13T03:32:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f46b4114250 con 0x7f46b4077bc0 2024-04-13T03:32:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b4078560 0x7f46b41a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46b4076980 0x7f46b419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b4078560 0x7f46b41a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36836/0 (socket says 172.21.15.16:36836) 2024-04-13T03:32:01.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3400640 1 -- 172.21.15.16:0/2997625489 learned_addr learned my addr 172.21.15.16:0/2997625489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:01.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46b4076980 0x7f46b419e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:36828/0 (socket says 172.21.15.16:36828) 2024-04-13T03:32:01.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46abe00640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b4077bc0 0x7f46b419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:01.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3400640 1 -- 172.21.15.16:0/2997625489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b4077bc0 msgr2=0x7f46b419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:01.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3400640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b4077bc0 0x7f46b419eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3400640 1 -- 172.21.15.16:0/2997625489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46b4076980 msgr2=0x7f46b419e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:01.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.822+0000 7f46b3400640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46b4076980 0x7f46b419e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:01.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.823+0000 7f46b3400640 1 -- 172.21.15.16:0/2997625489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b41a6380 con 0x7f46b4078560 2024-04-13T03:32:01.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.823+0000 7f46b3400640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b4078560 0x7f46b41a5d10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00eb10 tx=0x7f46ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:01.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.823+0000 7f46ab400640 1 -- 172.21.15.16:0/2997625489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ac004b20 con 0x7f46b4078560 2024-04-13T03:32:01.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.823+0000 7f46ab400640 1 -- 172.21.15.16:0/2997625489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46ac004cc0 con 0x7f46b4078560 2024-04-13T03:32:01.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.823+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b41a6670 con 0x7f46b4078560 2024-04-13T03:32:01.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.823+0000 7f46ab400640 1 -- 172.21.15.16:0/2997625489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ac010760 con 0x7f46b4078560 2024-04-13T03:32:01.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.823+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b41a6b40 con 0x7f46b4078560 2024-04-13T03:32:01.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.824+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b4110b90 con 0x7f46b4078560 2024-04-13T03:32:01.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.827+0000 7f46ab400640 1 -- 172.21.15.16:0/2997625489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f46ac005000 con 0x7f46b4078560 2024-04-13T03:32:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.828+0000 7f46ab400640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f46980a3bb0 0x7f46980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.828+0000 7f46ab400640 1 -- 172.21.15.16:0/2997625489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f46ac0d0e30 con 0x7f46b4078560 2024-04-13T03:32:01.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.828+0000 7f46b2a00640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f46980a3bb0 0x7f46980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:01.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.829+0000 7f46b2a00640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f46980a3bb0 0x7f46980a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f46a4005fd0 tx=0x7f46a400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:01.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:01.830+0000 7f46ab400640 1 -- 172.21.15.16:0/2997625489 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46ac053ff0 con 0x7f46b4078560 2024-04-13T03:32:02.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.039+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46b4062bc0 con 0x7f46b4078560 2024-04-13T03:32:02.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.041+0000 7f46ab400640 1 -- 172.21.15.16:0/2997625489 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7f46ac09d570 con 0x7f46b4078560 2024-04-13T03:32:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:32:02.043 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:32:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f46980a3bb0 msgr2=0x7f46980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f46980a3bb0 0x7f46980a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f46a4005fd0 tx=0x7f46a400b390 comp rx=0 tx=0).stop 2024-04-13T03:32:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b4078560 msgr2=0x7f46b41a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:02.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b4078560 0x7f46b41a5d10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00eb10 tx=0x7f46ac00c2d0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 shutdown_connections 2024-04-13T03:32:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f46980a3bb0 0x7f46980a6070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46b4078560 0x7f46b41a5d10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f46b4077bc0 0x7f46b419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 --2- 172.21.15.16:0/2997625489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46b4076980 0x7f46b419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 >> 172.21.15.16:0/2997625489 conn(0x7f46b40ff5d0 msgr2=0x7f46b4100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 shutdown_connections 2024-04-13T03:32:02.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.044+0000 7f46b3e00640 1 -- 172.21.15.16:0/2997625489 wait complete. 2024-04-13T03:32:02.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-13T03:32:02.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:01 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1709625965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.288+0000 7fb2f7200640 1 -- 172.21.15.16:0/2510471942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f81096f0 msgr2=0x7fb2f8109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.288+0000 7fb2f7200640 1 --2- 172.21.15.16:0/2510471942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f81096f0 0x7fb2f8109b70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009900 tx=0x7fb2e402bc40 comp rx=0 tx=0).stop 2024-04-13T03:32:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.288+0000 7fb2f7200640 1 -- 172.21.15.16:0/2510471942 shutdown_connections 2024-04-13T03:32:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.288+0000 7fb2f7200640 1 --2- 172.21.15.16:0/2510471942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f810a0b0 0x7fb2f8110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.288+0000 7fb2f7200640 1 --2- 172.21.15.16:0/2510471942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f81096f0 0x7fb2f8109b70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.288+0000 7fb2f7200640 1 --2- 172.21.15.16:0/2510471942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f8108500 0x7fb2f8108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.288+0000 7fb2f7200640 1 -- 172.21.15.16:0/2510471942 >> 172.21.15.16:0/2510471942 conn(0x7fb2f8077490 msgr2=0x7fb2f80798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:02.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.289+0000 7fb2f7200640 1 -- 172.21.15.16:0/2510471942 shutdown_connections 2024-04-13T03:32:02.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.289+0000 7fb2f7200640 1 -- 172.21.15.16:0/2510471942 wait complete. 2024-04-13T03:32:02.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.290+0000 7fb2f7200640 1 Processor -- start 2024-04-13T03:32:02.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.290+0000 7fb2f7200640 1 -- start start 2024-04-13T03:32:02.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.290+0000 7fb2f7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f8108500 0x7fb2f819e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:02.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.290+0000 7fb2f7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f81096f0 0x7fb2f819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:02.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.290+0000 7fb2f7200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f810a0b0 0x7fb2f81a5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.290+0000 7fb2f7200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f8114190 con 0x7fb2f81096f0 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.290+0000 7fb2f7200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f8114010 con 0x7fb2f8108500 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.290+0000 7fb2f7200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f8114310 con 0x7fb2f810a0b0 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f8108500 0x7fb2f819e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f8108500 0x7fb2f819e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:36850/0 (socket says 172.21.15.16:36850) 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f81096f0 0x7fb2f819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f5e00640 1 -- 172.21.15.16:0/707992732 learned_addr learned my addr 172.21.15.16:0/707992732 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f6800640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f810a0b0 0x7fb2f81a5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f5e00640 1 -- 172.21.15.16:0/707992732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f810a0b0 msgr2=0x7fb2f81a5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:02.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f5e00640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f810a0b0 0x7fb2f81a5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f5e00640 1 -- 172.21.15.16:0/707992732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f81096f0 msgr2=0x7fb2f819ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f5e00640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f81096f0 0x7fb2f819ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.291+0000 7fb2f5e00640 1 -- 172.21.15.16:0/707992732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f81a63b0 con 0x7fb2f8108500 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.292+0000 7fb2f5e00640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f8108500 0x7fb2f819e790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800e9c0 tx=0x7fb2e800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.292+0000 7fb2eea00640 1 -- 172.21.15.16:0/707992732 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e800cd80 con 0x7fb2f8108500 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.292+0000 7fb2eea00640 1 -- 172.21.15.16:0/707992732 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2e80035f0 con 0x7fb2f8108500 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.292+0000 7fb2eea00640 1 -- 172.21.15.16:0/707992732 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8010690 con 0x7fb2f8108500 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.292+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f81a66a0 con 0x7fb2f8108500 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.292+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f81a6b70 con 0x7fb2f8108500 2024-04-13T03:32:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.293+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f810f820 con 0x7fb2f8108500 2024-04-13T03:32:02.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.295+0000 7fb2eea00640 1 -- 172.21.15.16:0/707992732 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb2e80037a0 con 0x7fb2f8108500 2024-04-13T03:32:02.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.296+0000 7fb2eea00640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2dc0a3c30 0x7fb2dc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:02.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.297+0000 7fb2eea00640 1 -- 172.21.15.16:0/707992732 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb2e80d0a00 con 0x7fb2f8108500 2024-04-13T03:32:02.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.297+0000 7fb2f5400640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2dc0a3c30 0x7fb2dc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:02.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.297+0000 7fb2f5400640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2dc0a3c30 0x7fb2dc0a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb2e402c7c0 tx=0x7fb2e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:02.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.301+0000 7fb2eea00640 1 -- 172.21.15.16:0/707992732 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2e809d140 con 0x7fb2f8108500 2024-04-13T03:32:02.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:01 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1709625965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:02.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:01 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1709625965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:02.555+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb2f8062bc0 con 0x7fb2f8108500 2024-04-13T03:32:03.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.000+0000 7fb2eea00640 1 -- 172.21.15.16:0/707992732 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fb2e809ca40 con 0x7fb2f8108500 2024-04-13T03:32:03.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2dc0a3c30 msgr2=0x7fb2dc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:03.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2dc0a3c30 0x7fb2dc0a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb2e402c7c0 tx=0x7fb2e4005fd0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f8108500 msgr2=0x7fb2f819e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:03.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f8108500 0x7fb2f819e790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800e9c0 tx=0x7fb2e800ee80 comp rx=0 tx=0).stop 2024-04-13T03:32:03.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 shutdown_connections 2024-04-13T03:32:03.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2dc0a3c30 0x7fb2dc0a60f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2f810a0b0 0x7fb2f81a5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2f81096f0 0x7fb2f819ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 --2- 172.21.15.16:0/707992732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2f8108500 0x7fb2f819e790 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 >> 172.21.15.16:0/707992732 conn(0x7fb2f8077490 msgr2=0x7fb2f810c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 shutdown_connections 2024-04-13T03:32:03.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.005+0000 7fb2f7200640 1 -- 172.21.15.16:0/707992732 wait complete. 2024-04-13T03:32:03.029 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-13T03:32:03.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-13T03:32:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.264+0000 7fd34c800640 1 -- 172.21.15.16:0/3833762737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd344102a80 msgr2=0x7fd344104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.264+0000 7fd34c800640 1 --2- 172.21.15.16:0/3833762737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd344102a80 0x7fd344104e70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd33c009ca0 tx=0x7fd33c02c0f0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.264+0000 7fd34c800640 1 -- 172.21.15.16:0/3833762737 shutdown_connections 2024-04-13T03:32:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.264+0000 7fd34c800640 1 --2- 172.21.15.16:0/3833762737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd344107ce0 0x7fd34410a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.264+0000 7fd34c800640 1 --2- 172.21.15.16:0/3833762737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3441053b0 0x7fd3441077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.264+0000 7fd34c800640 1 --2- 172.21.15.16:0/3833762737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd344102a80 0x7fd344104e70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.264+0000 7fd34c800640 1 -- 172.21.15.16:0/3833762737 >> 172.21.15.16:0/3833762737 conn(0x7fd3440fc470 msgr2=0x7fd3440fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:03.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.264+0000 7fd34c800640 1 -- 172.21.15.16:0/3833762737 shutdown_connections 2024-04-13T03:32:03.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.264+0000 7fd34c800640 1 -- 172.21.15.16:0/3833762737 wait complete. 2024-04-13T03:32:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.266+0000 7fd34c800640 1 Processor -- start 2024-04-13T03:32:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.266+0000 7fd34c800640 1 -- start start 2024-04-13T03:32:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.266+0000 7fd34c800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd344102a80 0x7fd34419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:03.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.266+0000 7fd34c800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3441053b0 0x7fd34419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.266+0000 7fd34c800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd344107ce0 0x7fd3441a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd34c800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd34410dc90 con 0x7fd344102a80 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd34c800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd34410db10 con 0x7fd344107ce0 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd34c800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd34410de10 con 0x7fd3441053b0 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd349400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3441053b0 0x7fd34419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd349e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd344102a80 0x7fd34419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd349400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3441053b0 0x7fd34419aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:49324/0 (socket says 172.21.15.16:49324) 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd349400640 1 -- 172.21.15.16:0/3385648636 learned_addr learned my addr 172.21.15.16:0/3385648636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd34a800640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd344107ce0 0x7fd3441a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd349400640 1 -- 172.21.15.16:0/3385648636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd344107ce0 msgr2=0x7fd3441a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd349400640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd344107ce0 0x7fd3441a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd349400640 1 -- 172.21.15.16:0/3385648636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd344102a80 msgr2=0x7fd34419a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd349400640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd344102a80 0x7fd34419a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:03.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd349400640 1 -- 172.21.15.16:0/3385648636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3441a21d0 con 0x7fd3441053b0 2024-04-13T03:32:03.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.267+0000 7fd34a800640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd344107ce0 0x7fd3441a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.268+0000 7fd349400640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3441053b0 0x7fd34419aaf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd33400e9f0 tx=0x7fd33400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.268+0000 7fd33aa00640 1 -- 172.21.15.16:0/3385648636 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33400ccc0 con 0x7fd3441053b0 2024-04-13T03:32:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.268+0000 7fd33aa00640 1 -- 172.21.15.16:0/3385648636 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd334005500 con 0x7fd3441053b0 2024-04-13T03:32:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.268+0000 7fd33aa00640 1 -- 172.21.15.16:0/3385648636 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334010690 con 0x7fd3441053b0 2024-04-13T03:32:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.268+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3441a24c0 con 0x7fd3441053b0 2024-04-13T03:32:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.268+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3441a29c0 con 0x7fd3441053b0 2024-04-13T03:32:03.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.269+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd344109830 con 0x7fd3441053b0 2024-04-13T03:32:03.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.272+0000 7fd33aa00640 1 -- 172.21.15.16:0/3385648636 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd3340046c0 con 0x7fd3441053b0 2024-04-13T03:32:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.273+0000 7fd33aa00640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3300a3bb0 0x7fd3300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:03.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.274+0000 7fd33aa00640 1 -- 172.21.15.16:0/3385648636 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd3340d09e0 con 0x7fd3441053b0 2024-04-13T03:32:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.274+0000 7fd349e00640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3300a3bb0 0x7fd3300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:03.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.275+0000 7fd349e00640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3300a3bb0 0x7fd3300a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd33c02c700 tx=0x7fd33c0323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:03.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.278+0000 7fd33aa00640 1 -- 172.21.15.16:0/3385648636 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd33409d120 con 0x7fd3441053b0 2024-04-13T03:32:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:02 smithi119 ceph-mon[35835]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-13T03:32:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:02 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2997625489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:02 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/707992732' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-13T03:32:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:02 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-13T03:32:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:02 smithi016 ceph-mon[31772]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-13T03:32:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:02 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2997625489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:02 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/707992732' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-13T03:32:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:02 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-13T03:32:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:02 smithi188 ceph-mon[36398]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-13T03:32:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:02 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2997625489' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:03.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:02 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/707992732' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-13T03:32:03.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:02 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-13T03:32:03.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:03.527+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fd344062bc0 con 0x7fd3441053b0 2024-04-13T03:32:04.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.012+0000 7fd33aa00640 1 -- 172.21.15.16:0/3385648636 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fd33409ca20 con 0x7fd3441053b0 2024-04-13T03:32:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3300a3bb0 msgr2=0x7fd3300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3300a3bb0 0x7fd3300a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd33c02c700 tx=0x7fd33c0323c0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3441053b0 msgr2=0x7fd34419aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3441053b0 0x7fd34419aaf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd33400e9f0 tx=0x7fd33400eeb0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 shutdown_connections 2024-04-13T03:32:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd3300a3bb0 0x7fd3300a6070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd344107ce0 0x7fd3441a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd3441053b0 0x7fd34419aaf0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 --2- 172.21.15.16:0/3385648636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd344102a80 0x7fd34419a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.016+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 >> 172.21.15.16:0/3385648636 conn(0x7fd3440fc470 msgr2=0x7fd3441022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.017+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 shutdown_connections 2024-04-13T03:32:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.017+0000 7fd34c800640 1 -- 172.21.15.16:0/3385648636 wait complete. 2024-04-13T03:32:04.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-13T03:32:04.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.255+0000 7efc96c00640 1 -- 172.21.15.16:0/1233000349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc90104100 msgr2=0x7efc90104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:04.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.255+0000 7efc96c00640 1 --2- 172.21.15.16:0/1233000349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc90104100 0x7efc90104520 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efc84009900 tx=0x7efc8402bc30 comp rx=0 tx=0).stop 2024-04-13T03:32:04.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.255+0000 7efc96c00640 1 -- 172.21.15.16:0/1233000349 shutdown_connections 2024-04-13T03:32:04.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.255+0000 7efc96c00640 1 --2- 172.21.15.16:0/1233000349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc90105cb0 0x7efc9010c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.255+0000 7efc96c00640 1 --2- 172.21.15.16:0/1233000349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc901052f0 0x7efc90105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.255+0000 7efc96c00640 1 --2- 172.21.15.16:0/1233000349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc90104100 0x7efc90104520 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.255+0000 7efc96c00640 1 -- 172.21.15.16:0/1233000349 >> 172.21.15.16:0/1233000349 conn(0x7efc900ff670 msgr2=0x7efc90101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:04.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.255+0000 7efc96c00640 1 -- 172.21.15.16:0/1233000349 shutdown_connections 2024-04-13T03:32:04.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.255+0000 7efc96c00640 1 -- 172.21.15.16:0/1233000349 wait complete. 2024-04-13T03:32:04.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.256+0000 7efc96c00640 1 Processor -- start 2024-04-13T03:32:04.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc96c00640 1 -- start start 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc96c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc90104100 0x7efc9019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc96c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc901052f0 0x7efc9019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc96c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc90105cb0 0x7efc901a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc96c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efc9010fdb0 con 0x7efc901052f0 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc96c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efc9010fc30 con 0x7efc90105cb0 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc96c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efc9010ff30 con 0x7efc90104100 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc94e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc901052f0 0x7efc9019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc94e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc901052f0 0x7efc9019a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36894/0 (socket says 172.21.15.16:36894) 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc94e00640 1 -- 172.21.15.16:0/1371498139 learned_addr learned my addr 172.21.15.16:0/1371498139 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc96200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc90105cb0 0x7efc901a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.257+0000 7efc95800640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc90104100 0x7efc9019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc94e00640 1 -- 172.21.15.16:0/1371498139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc90104100 msgr2=0x7efc9019a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc94e00640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc90104100 0x7efc9019a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc94e00640 1 -- 172.21.15.16:0/1371498139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc90105cb0 msgr2=0x7efc901a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:04.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc94e00640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc90105cb0 0x7efc901a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:04.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc94e00640 1 -- 172.21.15.16:0/1371498139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc901a2010 con 0x7efc901052f0 2024-04-13T03:32:04.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc96200640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc90105cb0 0x7efc901a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:04.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc94e00640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc901052f0 0x7efc9019a930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00e9f0 tx=0x7efc7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:04.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc8e000640 1 -- 172.21.15.16:0/1371498139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c00cd80 con 0x7efc901052f0 2024-04-13T03:32:04.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc8e000640 1 -- 172.21.15.16:0/1371498139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc7c010910 con 0x7efc901052f0 2024-04-13T03:32:04.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc901a2300 con 0x7efc901052f0 2024-04-13T03:32:04.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.258+0000 7efc8e000640 1 -- 172.21.15.16:0/1371498139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c018990 con 0x7efc901052f0 2024-04-13T03:32:04.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.259+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc901a2850 con 0x7efc901052f0 2024-04-13T03:32:04.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.260+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc9010b420 con 0x7efc901052f0 2024-04-13T03:32:04.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.262+0000 7efc8e000640 1 -- 172.21.15.16:0/1371498139 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efc7c010ac0 con 0x7efc901052f0 2024-04-13T03:32:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.264+0000 7efc8e000640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efc780a3bb0 0x7efc780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.264+0000 7efc8e000640 1 -- 172.21.15.16:0/1371498139 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7efc7c0d0960 con 0x7efc901052f0 2024-04-13T03:32:04.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.264+0000 7efc95800640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efc780a3bb0 0x7efc780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:04.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.265+0000 7efc95800640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efc780a3bb0 0x7efc780a6070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efc8402c170 tx=0x7efc8402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:04.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.269+0000 7efc8e000640 1 -- 172.21.15.16:0/1371498139 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc7c09cab0 con 0x7efc901052f0 2024-04-13T03:32:04.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:04 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-13T03:32:04.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:04 smithi119 ceph-mon[35835]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:32:04.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:04 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3385648636' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-13T03:32:04.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:04 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-13T03:32:04.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:04 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-13T03:32:04.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:04 smithi016 ceph-mon[31772]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:32:04.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:04 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3385648636' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-13T03:32:04.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:04 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-13T03:32:04.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:04 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-13T03:32:04.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:04 smithi188 ceph-mon[36398]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:32:04.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:04 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3385648636' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-13T03:32:04.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:04 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-13T03:32:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:04.518+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7efc9002d5e0 con 0x7efc901052f0 2024-04-13T03:32:05.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.026+0000 7efc8e000640 1 -- 172.21.15.16:0/1371498139 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7efc7c0a0db0 con 0x7efc901052f0 2024-04-13T03:32:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efc780a3bb0 msgr2=0x7efc780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efc780a3bb0 0x7efc780a6070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efc8402c170 tx=0x7efc8402df90 comp rx=0 tx=0).stop 2024-04-13T03:32:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc901052f0 msgr2=0x7efc9019a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc901052f0 0x7efc9019a930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00e9f0 tx=0x7efc7c00eeb0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 shutdown_connections 2024-04-13T03:32:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efc780a3bb0 0x7efc780a6070 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc90105cb0 0x7efc901a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc901052f0 0x7efc9019a930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 --2- 172.21.15.16:0/1371498139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc90104100 0x7efc9019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 >> 172.21.15.16:0/1371498139 conn(0x7efc900ff670 msgr2=0x7efc901085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 shutdown_connections 2024-04-13T03:32:05.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.031+0000 7efc96c00640 1 -- 172.21.15.16:0/1371498139 wait complete. 2024-04-13T03:32:05.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-13T03:32:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.277+0000 7fc6d3e00640 1 -- 172.21.15.16:0/3193438258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4104100 msgr2=0x7fc6d4104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.277+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/3193438258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4104100 0x7fc6d4104520 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009900 tx=0x7fc6c402bc30 comp rx=0 tx=0).stop 2024-04-13T03:32:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.277+0000 7fc6d3e00640 1 -- 172.21.15.16:0/3193438258 shutdown_connections 2024-04-13T03:32:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.277+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/3193438258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d4105cb0 0x7fc6d410c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.277+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/3193438258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d41052f0 0x7fc6d4105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.277+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/3193438258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4104100 0x7fc6d4104520 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.277+0000 7fc6d3e00640 1 -- 172.21.15.16:0/3193438258 >> 172.21.15.16:0/3193438258 conn(0x7fc6d40ff670 msgr2=0x7fc6d4101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.277+0000 7fc6d3e00640 1 -- 172.21.15.16:0/3193438258 shutdown_connections 2024-04-13T03:32:05.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.277+0000 7fc6d3e00640 1 -- 172.21.15.16:0/3193438258 wait complete. 2024-04-13T03:32:05.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.278+0000 7fc6d3e00640 1 Processor -- start 2024-04-13T03:32:05.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.278+0000 7fc6d3e00640 1 -- start start 2024-04-13T03:32:05.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d4104100 0x7fc6d419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:05.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d41052f0 0x7fc6d419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4105cb0 0x7fc6d41a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410fde0 con 0x7fc6d4104100 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410fc60 con 0x7fc6d4105cb0 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410ff60 con 0x7fc6d41052f0 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4105cb0 0x7fc6d41a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4105cb0 0x7fc6d41a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:36904/0 (socket says 172.21.15.16:36904) 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3400640 1 -- 172.21.15.16:0/2067357589 learned_addr learned my addr 172.21.15.16:0/2067357589 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d2a00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d4104100 0x7fc6d419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d2000640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d41052f0 0x7fc6d419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3400640 1 -- 172.21.15.16:0/2067357589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d41052f0 msgr2=0x7fc6d419a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.279+0000 7fc6d3400640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d41052f0 0x7fc6d419a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d3400640 1 -- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d4104100 msgr2=0x7fc6d419a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d3400640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d4104100 0x7fc6d419a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:05.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d3400640 1 -- 172.21.15.16:0/2067357589 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d41a2030 con 0x7fc6d4105cb0 2024-04-13T03:32:05.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d2000640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d41052f0 0x7fc6d419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:32:05.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d3400640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4105cb0 0x7fc6d41a19c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00eb10 tx=0x7fc6cc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:05.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d8a00640 1 -- 172.21.15.16:0/2067357589 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc004b20 con 0x7fc6d4105cb0 2024-04-13T03:32:05.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d8a00640 1 -- 172.21.15.16:0/2067357589 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc004cc0 con 0x7fc6d4105cb0 2024-04-13T03:32:05.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d8a00640 1 -- 172.21.15.16:0/2067357589 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc010760 con 0x7fc6d4105cb0 2024-04-13T03:32:05.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d41a2320 con 0x7fc6d4105cb0 2024-04-13T03:32:05.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.280+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d41a2820 con 0x7fc6d4105cb0 2024-04-13T03:32:05.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.281+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d410b420 con 0x7fc6d4105cb0 2024-04-13T03:32:05.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.284+0000 7fc6d8a00640 1 -- 172.21.15.16:0/2067357589 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc6cc010900 con 0x7fc6d4105cb0 2024-04-13T03:32:05.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.285+0000 7fc6d8a00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc6b80a3c30 0x7fc6b80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:05.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.285+0000 7fc6d8a00640 1 -- 172.21.15.16:0/2067357589 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc6cc0d0c60 con 0x7fc6d4105cb0 2024-04-13T03:32:05.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.286+0000 7fc6d2a00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc6b80a3c30 0x7fc6b80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:05.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.286+0000 7fc6d2a00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc6b80a3c30 0x7fc6b80a60f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc6c402c170 tx=0x7fc6c402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.287+0000 7fc6d8a00640 1 -- 172.21.15.16:0/2067357589 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6cc09d3a0 con 0x7fc6d4105cb0 2024-04-13T03:32:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:05 smithi119 ceph-mon[35835]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-13T03:32:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:05 smithi119 ceph-mon[35835]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-13T03:32:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:05 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-13T03:32:05.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:05 smithi119 ceph-mon[35835]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-13T03:32:05.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:05 smithi119 ceph-mon[35835]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:32:05.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:05 smithi119 ceph-mon[35835]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-13T03:32:05.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:05 smithi119 ceph-mon[35835]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-13T03:32:05.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:05 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1371498139' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-13T03:32:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:05 smithi016 ceph-mon[31772]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-13T03:32:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:05 smithi016 ceph-mon[31772]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-13T03:32:05.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:05 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-13T03:32:05.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:05 smithi016 ceph-mon[31772]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-13T03:32:05.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:05 smithi016 ceph-mon[31772]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:32:05.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:05 smithi016 ceph-mon[31772]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-13T03:32:05.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:05 smithi016 ceph-mon[31772]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-13T03:32:05.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:05 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1371498139' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-13T03:32:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:05 smithi188 ceph-mon[36398]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-13T03:32:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:05 smithi188 ceph-mon[36398]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-13T03:32:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:05 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-13T03:32:05.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:05 smithi188 ceph-mon[36398]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-13T03:32:05.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:05 smithi188 ceph-mon[36398]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-13T03:32:05.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:05 smithi188 ceph-mon[36398]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-13T03:32:05.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:05 smithi188 ceph-mon[36398]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-13T03:32:05.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:05 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1371498139' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-13T03:32:05.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:05.541+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fc6d402d5e0 con 0x7fc6d4105cb0 2024-04-13T03:32:06.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.049+0000 7fc6d8a00640 1 -- 172.21.15.16:0/2067357589 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7fc6cc09cca0 con 0x7fc6d4105cb0 2024-04-13T03:32:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.052+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc6b80a3c30 msgr2=0x7fc6b80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.052+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc6b80a3c30 0x7fc6b80a60f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc6c402c170 tx=0x7fc6c402df90 comp rx=0 tx=0).stop 2024-04-13T03:32:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.052+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4105cb0 msgr2=0x7fc6d41a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.052+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4105cb0 0x7fc6d41a19c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00eb10 tx=0x7fc6cc00c2d0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.053+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 shutdown_connections 2024-04-13T03:32:06.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.053+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc6b80a3c30 0x7fc6b80a60f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.053+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6d4105cb0 0x7fc6d41a19c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.053+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6d41052f0 0x7fc6d419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.053+0000 7fc6d3e00640 1 --2- 172.21.15.16:0/2067357589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d4104100 0x7fc6d419a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.053+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 >> 172.21.15.16:0/2067357589 conn(0x7fc6d40ff670 msgr2=0x7fc6d41085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:06.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.053+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 shutdown_connections 2024-04-13T03:32:06.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.053+0000 7fc6d3e00640 1 -- 172.21.15.16:0/2067357589 wait complete. 2024-04-13T03:32:06.085 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-04-13T03:32:06.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-13T03:32:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.291+0000 7f90aa800640 1 -- 172.21.15.16:0/1558903419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 msgr2=0x7f90a4107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.291+0000 7f90aa800640 1 --2- 172.21.15.16:0/1558903419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 0x7f90a4107670 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9090009900 tx=0x7f909002bc40 comp rx=0 tx=0).stop 2024-04-13T03:32:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.291+0000 7f90aa800640 1 -- 172.21.15.16:0/1558903419 shutdown_connections 2024-04-13T03:32:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.291+0000 7f90aa800640 1 --2- 172.21.15.16:0/1558903419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90a4107bb0 0x7f90a410a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.291+0000 7f90aa800640 1 --2- 172.21.15.16:0/1558903419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 0x7f90a4107670 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.291+0000 7f90aa800640 1 --2- 172.21.15.16:0/1558903419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a4102950 0x7f90a4104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.291+0000 7f90aa800640 1 -- 172.21.15.16:0/1558903419 >> 172.21.15.16:0/1558903419 conn(0x7f90a40fc360 msgr2=0x7f90a40fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.291+0000 7f90aa800640 1 -- 172.21.15.16:0/1558903419 shutdown_connections 2024-04-13T03:32:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.291+0000 7f90aa800640 1 -- 172.21.15.16:0/1558903419 wait complete. 2024-04-13T03:32:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.292+0000 7f90aa800640 1 Processor -- start 2024-04-13T03:32:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.292+0000 7f90aa800640 1 -- start start 2024-04-13T03:32:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90aa800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90a4102950 0x7f90a41981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90aa800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 0x7f90a4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:06.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90aa800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a4107bb0 0x7f90a419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90aa800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90a410db90 con 0x7f90a4107bb0 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90aa800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f90a410da10 con 0x7f90a4105280 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90aa800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f90a410dd10 con 0x7f90a4102950 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90a8a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 0x7f90a4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90a8a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 0x7f90a4198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:36930/0 (socket says 172.21.15.16:36930) 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90a8a00640 1 -- 172.21.15.16:0/3593219550 learned_addr learned my addr 172.21.15.16:0/3593219550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90a9400640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90a4102950 0x7f90a41981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.293+0000 7f90a9e00640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a4107bb0 0x7f90a419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90a8a00640 1 -- 172.21.15.16:0/3593219550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90a4102950 msgr2=0x7f90a41981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90a8a00640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90a4102950 0x7f90a41981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90a8a00640 1 -- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a4107bb0 msgr2=0x7f90a419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90a8a00640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a4107bb0 0x7f90a419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90a8a00640 1 -- 172.21.15.16:0/3593219550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a419fdf0 con 0x7f90a4105280 2024-04-13T03:32:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90a8a00640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 0x7f90a4198710 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f909002efd0 tx=0x7f9090005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90a2000640 1 -- 172.21.15.16:0/3593219550 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090041070 con 0x7f90a4105280 2024-04-13T03:32:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90a2000640 1 -- 172.21.15.16:0/3593219550 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90900054e0 con 0x7f90a4105280 2024-04-13T03:32:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90a41a0080 con 0x7f90a4105280 2024-04-13T03:32:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90a2000640 1 -- 172.21.15.16:0/3593219550 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9090034440 con 0x7f90a4105280 2024-04-13T03:32:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.294+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90a41a0540 con 0x7f90a4105280 2024-04-13T03:32:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.296+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90a40fde50 con 0x7f90a4105280 2024-04-13T03:32:06.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.297+0000 7f90a2000640 1 -- 172.21.15.16:0/3593219550 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9090044070 con 0x7f90a4105280 2024-04-13T03:32:06.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.299+0000 7f90a2000640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f908c0a3bb0 0x7f908c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.299+0000 7f90a2000640 1 -- 172.21.15.16:0/3593219550 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f90900f2330 con 0x7f90a4105280 2024-04-13T03:32:06.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.299+0000 7f90a9400640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f908c0a3bb0 0x7f908c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:06.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.300+0000 7f90a9400640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f908c0a3bb0 0x7f908c0a6070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f909800bbe0 tx=0x7f909800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:06.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.302+0000 7f90a2000640 1 -- 172.21.15.16:0/3593219550 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90900bea70 con 0x7f90a4105280 2024-04-13T03:32:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:06 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1371498139' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-13T03:32:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] up:active 2024-04-13T03:32:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:06 smithi119 ceph-mon[35835]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-13T03:32:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:06 smithi119 ceph-mon[35835]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-13T03:32:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:06 smithi119 ceph-mon[35835]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-13T03:32:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:06 smithi119 ceph-mon[35835]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-13T03:32:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:06 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2067357589' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-13T03:32:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:06 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-13T03:32:06.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1371498139' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-13T03:32:06.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] up:active 2024-04-13T03:32:06.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-mon[31772]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-13T03:32:06.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-mon[31772]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-13T03:32:06.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-mon[31772]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-13T03:32:06.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-mon[31772]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-13T03:32:06.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2067357589' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-13T03:32:06.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-13T03:32:06.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:06 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1371498139' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-13T03:32:06.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] up:active 2024-04-13T03:32:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:06 smithi188 ceph-mon[36398]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-13T03:32:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:06 smithi188 ceph-mon[36398]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-13T03:32:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:06 smithi188 ceph-mon[36398]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-13T03:32:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:06 smithi188 ceph-mon[36398]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-13T03:32:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:06 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2067357589' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-13T03:32:06.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:06 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-13T03:32:06.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.541+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f90a4062400 con 0x7f908c0a3bb0 2024-04-13T03:32:06.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.580+0000 7f90a2000640 1 -- 172.21.15.16:0/3593219550 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f90a4062400 con 0x7f908c0a3bb0 2024-04-13T03:32:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f908c0a3bb0 msgr2=0x7f908c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f908c0a3bb0 0x7f908c0a6070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f909800bbe0 tx=0x7f909800b390 comp rx=0 tx=0).stop 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 msgr2=0x7f90a4198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 0x7f90a4198710 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f909002efd0 tx=0x7f9090005e40 comp rx=0 tx=0).stop 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 shutdown_connections 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f908c0a3bb0 0x7f908c0a6070 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90a4107bb0 0x7f90a419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a4105280 0x7f90a4198710 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 --2- 172.21.15.16:0/3593219550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90a4102950 0x7f90a41981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 >> 172.21.15.16:0/3593219550 conn(0x7f90a40fc360 msgr2=0x7f90a4103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 shutdown_connections 2024-04-13T03:32:06.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.583+0000 7f90aa800640 1 -- 172.21.15.16:0/3593219550 wait complete. 2024-04-13T03:32:06.603 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-13T03:32:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.819+0000 7f2858a00640 1 -- 172.21.15.16:0/2695266465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854103a40 msgr2=0x7f285410a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.819+0000 7f2858a00640 1 --2- 172.21.15.16:0/2695266465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854103a40 0x7f285410a360 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f284c009900 tx=0x7f284c02bc30 comp rx=0 tx=0).stop 2024-04-13T03:32:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.819+0000 7f2858a00640 1 -- 172.21.15.16:0/2695266465 shutdown_connections 2024-04-13T03:32:06.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.819+0000 7f2858a00640 1 --2- 172.21.15.16:0/2695266465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854103a40 0x7f285410a360 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.819+0000 7f2858a00640 1 --2- 172.21.15.16:0/2695266465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2854103080 0x7f2854103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.819+0000 7f2858a00640 1 --2- 172.21.15.16:0/2695266465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2854101ea0 0x7f28541022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.819+0000 7f2858a00640 1 -- 172.21.15.16:0/2695266465 >> 172.21.15.16:0/2695266465 conn(0x7f28540fd460 msgr2=0x7f28540ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.819+0000 7f2858a00640 1 -- 172.21.15.16:0/2695266465 shutdown_connections 2024-04-13T03:32:06.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.819+0000 7f2858a00640 1 -- 172.21.15.16:0/2695266465 wait complete. 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.820+0000 7f2858a00640 1 Processor -- start 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.820+0000 7f2858a00640 1 -- start start 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2858a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854101ea0 0x7f285419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2858a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2854103080 0x7f285419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2858a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2854103a40 0x7f28541a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2858a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f285410dbe0 con 0x7f2854103a40 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2858a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f285410da60 con 0x7f2854101ea0 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2858a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f285410dd60 con 0x7f2854103080 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2852800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854101ea0 0x7f285419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:06.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2852800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854101ea0 0x7f285419a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:36970/0 (socket says 172.21.15.16:36970) 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2852800640 1 -- 172.21.15.16:0/2907077426 learned_addr learned my addr 172.21.15.16:0/2907077426 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2849e00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2854103080 0x7f285419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2853200640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2854103a40 0x7f28541a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2852800640 1 -- 172.21.15.16:0/2907077426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2854103080 msgr2=0x7f285419a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2852800640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2854103080 0x7f285419a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2852800640 1 -- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2854103a40 msgr2=0x7f28541a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2852800640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2854103a40 0x7f28541a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.821+0000 7f2852800640 1 -- 172.21.15.16:0/2907077426 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28541a1fa0 con 0x7f2854101ea0 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.822+0000 7f2852800640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854101ea0 0x7f285419a380 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f284400eb70 tx=0x7f284400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.822+0000 7f284be00640 1 -- 172.21.15.16:0/2907077426 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844003b70 con 0x7f2854101ea0 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.822+0000 7f284be00640 1 -- 172.21.15.16:0/2907077426 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f284400ee70 con 0x7f2854101ea0 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.822+0000 7f284be00640 1 -- 172.21.15.16:0/2907077426 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844010830 con 0x7f2854101ea0 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.822+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28541a2290 con 0x7f2854101ea0 2024-04-13T03:32:06.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.822+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28541a2790 con 0x7f2854101ea0 2024-04-13T03:32:06.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.823+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28541a2a50 con 0x7f2854101ea0 2024-04-13T03:32:06.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.825+0000 7f284be00640 1 -- 172.21.15.16:0/2907077426 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2844004020 con 0x7f2854101ea0 2024-04-13T03:32:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.826+0000 7f284be00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f28380a3bb0 0x7f28380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:06.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.827+0000 7f284be00640 1 -- 172.21.15.16:0/2907077426 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f28440d0d20 con 0x7f2854101ea0 2024-04-13T03:32:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.827+0000 7f2849e00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f28380a3bb0 0x7f28380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:06.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.827+0000 7f2849e00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f28380a3bb0 0x7f28380a6070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f285419b890 tx=0x7f283c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:06.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:06.830+0000 7f284be00640 1 -- 172.21.15.16:0/2907077426 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f284409d460 con 0x7f2854101ea0 2024-04-13T03:32:06.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:32:06.553+0000 7f7a3d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:32:06.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:32:06.553+0000 7f7a3d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:32:06.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:32:06.553+0000 7f7a3d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:32:06.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:32:06.553+0000 7f7a3d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:32:06.855 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:32:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:32:06.553+0000 7f7a3d200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:32:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.068+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2854075910 con 0x7f28380a3bb0 2024-04-13T03:32:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.110+0000 7f284be00640 1 -- 172.21.15.16:0/2907077426 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2854075910 con 0x7f28380a3bb0 2024-04-13T03:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f28380a3bb0 msgr2=0x7f28380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f28380a3bb0 0x7f28380a6070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f285419b890 tx=0x7f283c005f90 comp rx=0 tx=0).stop 2024-04-13T03:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854101ea0 msgr2=0x7f285419a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854101ea0 0x7f285419a380 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f284400eb70 tx=0x7f284400c6a0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 shutdown_connections 2024-04-13T03:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f28380a3bb0 0x7f28380a6070 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2854103a40 0x7f28541a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2854103080 0x7f285419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 --2- 172.21.15.16:0/2907077426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2854101ea0 0x7f285419a380 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 >> 172.21.15.16:0/2907077426 conn(0x7f28540fd460 msgr2=0x7f28540741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 shutdown_connections 2024-04-13T03:32:07.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.113+0000 7f2858a00640 1 -- 172.21.15.16:0/2907077426 wait complete. 2024-04-13T03:32:07.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-13T03:32:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-mon[35835]: pgmap v333: 129 pgs: 129 active+clean; 582 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 1 op/s 2024-04-13T03:32:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-13T03:32:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] up:active 2024-04-13T03:32:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-mon[35835]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-13T03:32:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-mon[35835]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-13T03:32:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-mon[35835]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-04-13T03:32:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-mon[35835]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-13T03:32:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-mon[35835]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:07 smithi016 ceph-mon[31772]: pgmap v333: 129 pgs: 129 active+clean; 582 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 1 op/s 2024-04-13T03:32:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:07 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-13T03:32:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:07 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] up:active 2024-04-13T03:32:07.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:07 smithi016 ceph-mon[31772]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-13T03:32:07.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:07 smithi016 ceph-mon[31772]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-13T03:32:07.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:07 smithi016 ceph-mon[31772]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-04-13T03:32:07.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:07 smithi016 ceph-mon[31772]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-13T03:32:07.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:07 smithi016 ceph-mon[31772]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:07.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.356+0000 7f1950c00640 1 -- 172.21.15.16:0/812098198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f194c108500 msgr2=0x7f194c108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.356+0000 7f1950c00640 1 --2- 172.21.15.16:0/812098198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f194c108500 0x7f194c108920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f193c009900 tx=0x7f193c02bc30 comp rx=0 tx=0).stop 2024-04-13T03:32:07.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.356+0000 7f1950c00640 1 -- 172.21.15.16:0/812098198 shutdown_connections 2024-04-13T03:32:07.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.356+0000 7f1950c00640 1 --2- 172.21.15.16:0/812098198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194c10a0b0 0x7f194c110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.356+0000 7f1950c00640 1 --2- 172.21.15.16:0/812098198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f194c1096f0 0x7f194c109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.356+0000 7f1950c00640 1 --2- 172.21.15.16:0/812098198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f194c108500 0x7f194c108920 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.356+0000 7f1950c00640 1 -- 172.21.15.16:0/812098198 >> 172.21.15.16:0/812098198 conn(0x7f194c077490 msgr2=0x7f194c0798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:07.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.356+0000 7f1950c00640 1 -- 172.21.15.16:0/812098198 shutdown_connections 2024-04-13T03:32:07.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.356+0000 7f1950c00640 1 -- 172.21.15.16:0/812098198 wait complete. 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.357+0000 7f1950c00640 1 Processor -- start 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.357+0000 7f1950c00640 1 -- start start 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.357+0000 7f1950c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194c108500 0x7f194c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.357+0000 7f1950c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f194c1096f0 0x7f194c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f1950c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f194c10a0b0 0x7f194c1a5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f1950c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f194c1141e0 con 0x7f194c108500 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f1950c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f194c114060 con 0x7f194c1096f0 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f1950c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f194c114360 con 0x7f194c10a0b0 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194c108500 0x7f194c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194b400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194c108500 0x7f194c19e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36976/0 (socket says 172.21.15.16:36976) 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194b400640 1 -- 172.21.15.16:0/3717136797 learned_addr learned my addr 172.21.15.16:0/3717136797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:07.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194aa00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f194c1096f0 0x7f194c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:07.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194be00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f194c10a0b0 0x7f194c1a5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:07.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194b400640 1 -- 172.21.15.16:0/3717136797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f194c10a0b0 msgr2=0x7f194c1a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194b400640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f194c10a0b0 0x7f194c1a5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194b400640 1 -- 172.21.15.16:0/3717136797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f194c1096f0 msgr2=0x7f194c19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194b400640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f194c1096f0 0x7f194c19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194b400640 1 -- 172.21.15.16:0/3717136797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194c1a6490 con 0x7f194c108500 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194be00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f194c10a0b0 0x7f194c1a5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.358+0000 7f194aa00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f194c1096f0 0x7f194c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.359+0000 7f194b400640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194c108500 0x7f194c19e870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f193c005ab0 tx=0x7f193c02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.359+0000 7f1943e00640 1 -- 172.21.15.16:0/3717136797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c043020 con 0x7f194c108500 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.359+0000 7f1943e00640 1 -- 172.21.15.16:0/3717136797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f193c0054e0 con 0x7f194c108500 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.359+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f194c1a6720 con 0x7f194c108500 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.359+0000 7f1943e00640 1 -- 172.21.15.16:0/3717136797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c003ce0 con 0x7f194c108500 2024-04-13T03:32:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.359+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f194c1a6c70 con 0x7f194c108500 2024-04-13T03:32:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.361+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f194c10f7c0 con 0x7f194c108500 2024-04-13T03:32:07.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.363+0000 7f1943e00640 1 -- 172.21.15.16:0/3717136797 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f193c046020 con 0x7f194c108500 2024-04-13T03:32:07.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.365+0000 7f1943e00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19300a3bb0 0x7f19300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:07.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.365+0000 7f1943e00640 1 -- 172.21.15.16:0/3717136797 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f193c0f2810 con 0x7f194c108500 2024-04-13T03:32:07.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.366+0000 7f194aa00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19300a3bb0 0x7f19300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:07.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.366+0000 7f194aa00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19300a3bb0 0x7f19300a6070 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f194c19fd80 tx=0x7f1938005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:07.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.371+0000 7f1943e00640 1 -- 172.21.15.16:0/3717136797 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f193c04d020 con 0x7f194c108500 2024-04-13T03:32:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:07 smithi188 ceph-mon[36398]: pgmap v333: 129 pgs: 129 active+clean; 582 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 1 op/s 2024-04-13T03:32:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:07 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-13T03:32:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:07 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] up:active 2024-04-13T03:32:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:07 smithi188 ceph-mon[36398]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-13T03:32:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:07 smithi188 ceph-mon[36398]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-13T03:32:07.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:07 smithi188 ceph-mon[36398]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-04-13T03:32:07.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:07 smithi188 ceph-mon[36398]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-13T03:32:07.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:07 smithi188 ceph-mon[36398]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.612+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f194c062490 con 0x7f19300a3bb0 2024-04-13T03:32:07.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.624+0000 7f1943e00640 1 -- 172.21.15.16:0/3717136797 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f194c062490 con 0x7f19300a3bb0 2024-04-13T03:32:07.625 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_0/59e224ef-6f59-4692-92a8-12ebdb102d59 2024-04-13T03:32:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19300a3bb0 msgr2=0x7f19300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19300a3bb0 0x7f19300a6070 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f194c19fd80 tx=0x7f1938005f90 comp rx=0 tx=0).stop 2024-04-13T03:32:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194c108500 msgr2=0x7f194c19e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194c108500 0x7f194c19e870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f193c005ab0 tx=0x7f193c02c6c0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 shutdown_connections 2024-04-13T03:32:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f19300a3bb0 0x7f19300a6070 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f194c10a0b0 0x7f194c1a5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f194c1096f0 0x7f194c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 --2- 172.21.15.16:0/3717136797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f194c108500 0x7f194c19e870 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 >> 172.21.15.16:0/3717136797 conn(0x7f194c077490 msgr2=0x7f194c10c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 shutdown_connections 2024-04-13T03:32:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.627+0000 7f1950c00640 1 -- 172.21.15.16:0/3717136797 wait complete. 2024-04-13T03:32:07.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-13T03:32:07.821 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: 2024-04-13T03:32:07.512+0000 7f7758253700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-13T03:32:07.821 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: 2024-04-13T03:32:07.512+0000 7f7758253700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-13T03:32:07.822 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:32:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: 2024-04-13T03:32:07.512+0000 7f7758253700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-13T03:32:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.887+0000 7fcf6a600640 1 -- 172.21.15.16:0/2320312132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf641052f0 msgr2=0x7fcf64105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.887+0000 7fcf6a600640 1 --2- 172.21.15.16:0/2320312132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf641052f0 0x7fcf64105770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009900 tx=0x7fcf4c02bc30 comp rx=0 tx=0).stop 2024-04-13T03:32:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.887+0000 7fcf6a600640 1 -- 172.21.15.16:0/2320312132 shutdown_connections 2024-04-13T03:32:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.887+0000 7fcf6a600640 1 --2- 172.21.15.16:0/2320312132 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf64105cb0 0x7fcf6410c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.887+0000 7fcf6a600640 1 --2- 172.21.15.16:0/2320312132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf641052f0 0x7fcf64105770 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.887+0000 7fcf6a600640 1 --2- 172.21.15.16:0/2320312132 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf64104100 0x7fcf64104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.887+0000 7fcf6a600640 1 -- 172.21.15.16:0/2320312132 >> 172.21.15.16:0/2320312132 conn(0x7fcf640ff670 msgr2=0x7fcf64101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.887+0000 7fcf6a600640 1 -- 172.21.15.16:0/2320312132 shutdown_connections 2024-04-13T03:32:07.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.887+0000 7fcf6a600640 1 -- 172.21.15.16:0/2320312132 wait complete. 2024-04-13T03:32:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.888+0000 7fcf6a600640 1 Processor -- start 2024-04-13T03:32:07.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.888+0000 7fcf6a600640 1 -- start start 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.888+0000 7fcf6a600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf64104100 0x7fcf6419a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.888+0000 7fcf6a600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf641052f0 0x7fcf6419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.888+0000 7fcf6a600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf64105cb0 0x7fcf641a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.888+0000 7fcf6a600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6410fde0 con 0x7fcf641052f0 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.888+0000 7fcf6a600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6410fc60 con 0x7fcf64104100 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.888+0000 7fcf6a600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6410ff60 con 0x7fcf64105cb0 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf64104100 0x7fcf6419a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf64104100 0x7fcf6419a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:37006/0 (socket says 172.21.15.16:37006) 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69200640 1 -- 172.21.15.16:0/1677712851 learned_addr learned my addr 172.21.15.16:0/1677712851 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69c00640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf64105cb0 0x7fcf641a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf63e00640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf641052f0 0x7fcf6419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69200640 1 -- 172.21.15.16:0/1677712851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf64105cb0 msgr2=0x7fcf641a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69200640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf64105cb0 0x7fcf641a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69200640 1 -- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf641052f0 msgr2=0x7fcf6419a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69200640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf641052f0 0x7fcf6419a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69200640 1 -- 172.21.15.16:0/1677712851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf641a2010 con 0x7fcf64104100 2024-04-13T03:32:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.889+0000 7fcf69200640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf64104100 0x7fcf6419a3f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ebc0 tx=0x7fcf5400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.890+0000 7fcf61600640 1 -- 172.21.15.16:0/1677712851 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54003b70 con 0x7fcf64104100 2024-04-13T03:32:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.890+0000 7fcf61600640 1 -- 172.21.15.16:0/1677712851 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf54003d10 con 0x7fcf64104100 2024-04-13T03:32:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.890+0000 7fcf61600640 1 -- 172.21.15.16:0/1677712851 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54019980 con 0x7fcf64104100 2024-04-13T03:32:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.890+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf641a2300 con 0x7fcf64104100 2024-04-13T03:32:07.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.890+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf641a2870 con 0x7fcf64104100 2024-04-13T03:32:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.891+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf6410b420 con 0x7fcf64104100 2024-04-13T03:32:07.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.893+0000 7fcf61600640 1 -- 172.21.15.16:0/1677712851 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcf54019b20 con 0x7fcf64104100 2024-04-13T03:32:07.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.894+0000 7fcf61600640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf480a3c30 0x7fcf480a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:07.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.894+0000 7fcf61600640 1 -- 172.21.15.16:0/1677712851 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcf540d0e20 con 0x7fcf64104100 2024-04-13T03:32:07.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.895+0000 7fcf63e00640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf480a3c30 0x7fcf480a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:07.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.895+0000 7fcf63e00640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf480a3c30 0x7fcf480a60f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c02c240 tx=0x7fcf4c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:07.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:07.898+0000 7fcf61600640 1 -- 172.21.15.16:0/1677712851 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcf5409d560 con 0x7fcf64104100 2024-04-13T03:32:08.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.139+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fcf64062400 con 0x7fcf480a3c30 2024-04-13T03:32:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.179+0000 7fcf61600640 1 -- 172.21.15.16:0/1677712851 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcf64062400 con 0x7fcf480a3c30 2024-04-13T03:32:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.181+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf480a3c30 msgr2=0x7fcf480a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.181+0000 7fcf6a600640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf480a3c30 0x7fcf480a60f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c02c240 tx=0x7fcf4c02df90 comp rx=0 tx=0).stop 2024-04-13T03:32:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.181+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf64104100 msgr2=0x7fcf6419a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.181+0000 7fcf6a600640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf64104100 0x7fcf6419a3f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ebc0 tx=0x7fcf5400c6a0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.182+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 shutdown_connections 2024-04-13T03:32:08.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.182+0000 7fcf6a600640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcf480a3c30 0x7fcf480a60f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.182+0000 7fcf6a600640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf64105cb0 0x7fcf641a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.182+0000 7fcf6a600640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcf641052f0 0x7fcf6419a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.182+0000 7fcf6a600640 1 --2- 172.21.15.16:0/1677712851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf64104100 0x7fcf6419a3f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.182+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 >> 172.21.15.16:0/1677712851 conn(0x7fcf640ff670 msgr2=0x7fcf641085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.182+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 shutdown_connections 2024-04-13T03:32:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.182+0000 7fcf6a600640 1 -- 172.21.15.16:0/1677712851 wait complete. 2024-04-13T03:32:08.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-13T03:32:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] up:active 2024-04-13T03:32:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-13T03:32:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-13T03:32:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: Cluster is now healthy 2024-04-13T03:32:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-04-13T03:32:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:08.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-13T03:32:08.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-04-13T03:32:08.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] up:active 2024-04-13T03:32:08.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:08 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:32:08.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] up:active 2024-04-13T03:32:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-13T03:32:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-13T03:32:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: Cluster is now healthy 2024-04-13T03:32:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-04-13T03:32:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-13T03:32:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-04-13T03:32:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] up:active 2024-04-13T03:32:08.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:08 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:32:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] up:active 2024-04-13T03:32:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-13T03:32:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-13T03:32:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: Cluster is now healthy 2024-04-13T03:32:08.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-04-13T03:32:08.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:08.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-13T03:32:08.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-04-13T03:32:08.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] up:active 2024-04-13T03:32:08.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:08 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:32:08.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.440+0000 7fa52aa00640 1 -- 172.21.15.16:0/3071986443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa524103080 msgr2=0x7fa524103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.440+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3071986443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa524103080 0x7fa524103500 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa510009900 tx=0x7fa51002bc30 comp rx=0 tx=0).stop 2024-04-13T03:32:08.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.440+0000 7fa52aa00640 1 -- 172.21.15.16:0/3071986443 shutdown_connections 2024-04-13T03:32:08.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.440+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3071986443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa524103a40 0x7fa52410a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.440+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3071986443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa524103080 0x7fa524103500 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.440+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3071986443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa524101ea0 0x7fa5241022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.440+0000 7fa52aa00640 1 -- 172.21.15.16:0/3071986443 >> 172.21.15.16:0/3071986443 conn(0x7fa5240fd460 msgr2=0x7fa5240ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:08.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.440+0000 7fa52aa00640 1 -- 172.21.15.16:0/3071986443 shutdown_connections 2024-04-13T03:32:08.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.440+0000 7fa52aa00640 1 -- 172.21.15.16:0/3071986443 wait complete. 2024-04-13T03:32:08.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.441+0000 7fa52aa00640 1 Processor -- start 2024-04-13T03:32:08.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.441+0000 7fa52aa00640 1 -- start start 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.441+0000 7fa52aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa524101ea0 0x7fa524075d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa52aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa524103080 0x7fa5240743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa52aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa524103a40 0x7fa5240748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa52aa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa52410dbe0 con 0x7fa524101ea0 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa52aa00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa52410da60 con 0x7fa524103080 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa52aa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa52410dd60 con 0x7fa524103a40 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa529600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa524101ea0 0x7fa524075d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa529600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa524101ea0 0x7fa524075d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36994/0 (socket says 172.21.15.16:36994) 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa529600640 1 -- 172.21.15.16:0/3286892159 learned_addr learned my addr 172.21.15.16:0/3286892159 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa528c00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa524103080 0x7fa5240743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa52a000640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa524103a40 0x7fa5240748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa529600640 1 -- 172.21.15.16:0/3286892159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa524103a40 msgr2=0x7fa5240748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa529600640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa524103a40 0x7fa5240748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa529600640 1 -- 172.21.15.16:0/3286892159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa524103080 msgr2=0x7fa5240743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa529600640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa524103080 0x7fa5240743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa529600640 1 -- 172.21.15.16:0/3286892159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa524075180 con 0x7fa524101ea0 2024-04-13T03:32:08.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.442+0000 7fa528c00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa524103080 0x7fa5240743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:08.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.443+0000 7fa52a000640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa524103a40 0x7fa5240748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:32:08.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.443+0000 7fa529600640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa524101ea0 0x7fa524075d70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa51800eb70 tx=0x7fa51800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:08.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.443+0000 7fa522000640 1 -- 172.21.15.16:0/3286892159 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518003b70 con 0x7fa524101ea0 2024-04-13T03:32:08.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.443+0000 7fa522000640 1 -- 172.21.15.16:0/3286892159 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa51800ee70 con 0x7fa524101ea0 2024-04-13T03:32:08.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.443+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5241a2310 con 0x7fa524101ea0 2024-04-13T03:32:08.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.443+0000 7fa522000640 1 -- 172.21.15.16:0/3286892159 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518010830 con 0x7fa524101ea0 2024-04-13T03:32:08.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.443+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5241a27e0 con 0x7fa524101ea0 2024-04-13T03:32:08.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.445+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa524068d00 con 0x7fa524101ea0 2024-04-13T03:32:08.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.447+0000 7fa522000640 1 -- 172.21.15.16:0/3286892159 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa518004020 con 0x7fa524101ea0 2024-04-13T03:32:08.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.448+0000 7fa522000640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa50c0a3bb0 0x7fa50c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:08.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.448+0000 7fa522000640 1 -- 172.21.15.16:0/3286892159 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa5180d0e50 con 0x7fa524101ea0 2024-04-13T03:32:08.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.449+0000 7fa528c00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa50c0a3bb0 0x7fa50c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:08.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.450+0000 7fa528c00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa50c0a3bb0 0x7fa50c0a6070 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa51002c240 tx=0x7fa51002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:08.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.453+0000 7fa522000640 1 -- 172.21.15.16:0/3286892159 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa51809d590 con 0x7fa524101ea0 2024-04-13T03:32:08.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.698+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fa524062400 con 0x7fa50c0a3bb0 2024-04-13T03:32:08.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.711+0000 7fa522000640 1 -- 172.21.15.16:0/3286892159 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa524062400 con 0x7fa50c0a3bb0 2024-04-13T03:32:08.712 INFO:teuthology.orchestra.run.smithi016.stdout:/volumes/qa/sv_1/0eee58cc-391c-4f12-b842-2f4e801dfa8e 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.713+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa50c0a3bb0 msgr2=0x7fa50c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.713+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa50c0a3bb0 0x7fa50c0a6070 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa51002c240 tx=0x7fa51002df90 comp rx=0 tx=0).stop 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.713+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa524101ea0 msgr2=0x7fa524075d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.713+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa524101ea0 0x7fa524075d70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa51800eb70 tx=0x7fa51800c6a0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.713+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 shutdown_connections 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.713+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa50c0a3bb0 0x7fa50c0a6070 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.713+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa524103a40 0x7fa5240748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.713+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa524103080 0x7fa5240743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.713+0000 7fa52aa00640 1 --2- 172.21.15.16:0/3286892159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa524101ea0 0x7fa524075d70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.714+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 >> 172.21.15.16:0/3286892159 conn(0x7fa5240fd460 msgr2=0x7fa5241063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.714+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 shutdown_connections 2024-04-13T03:32:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.714+0000 7fa52aa00640 1 -- 172.21.15.16:0/3286892159 wait complete. 2024-04-13T03:32:08.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:32:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.961+0000 7f198ea00640 1 -- 172.21.15.16:0/846038992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1988109810 msgr2=0x7f1988109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.961+0000 7f198ea00640 1 --2- 172.21.15.16:0/846038992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1988109810 0x7f1988109c90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1974009900 tx=0x7f197402bc30 comp rx=0 tx=0).stop 2024-04-13T03:32:08.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.961+0000 7f198ea00640 1 -- 172.21.15.16:0/846038992 shutdown_connections 2024-04-13T03:32:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.961+0000 7f198ea00640 1 --2- 172.21.15.16:0/846038992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f198810a1d0 0x7f1988110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.961+0000 7f198ea00640 1 --2- 172.21.15.16:0/846038992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1988109810 0x7f1988109c90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.961+0000 7f198ea00640 1 --2- 172.21.15.16:0/846038992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1988108620 0x7f1988108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.961+0000 7f198ea00640 1 -- 172.21.15.16:0/846038992 >> 172.21.15.16:0/846038992 conn(0x7f19880775e0 msgr2=0x7f1988079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.961+0000 7f198ea00640 1 -- 172.21.15.16:0/846038992 shutdown_connections 2024-04-13T03:32:08.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.961+0000 7f198ea00640 1 -- 172.21.15.16:0/846038992 wait complete. 2024-04-13T03:32:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.962+0000 7f198ea00640 1 Processor -- start 2024-04-13T03:32:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ea00640 1 -- start start 2024-04-13T03:32:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1988108620 0x7f198819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1988109810 0x7f198819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f198810a1d0 0x7f19881a5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ea00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1988114300 con 0x7f198810a1d0 2024-04-13T03:32:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ea00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1988114180 con 0x7f1988109810 2024-04-13T03:32:08.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ea00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1988114480 con 0x7f1988108620 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f198810a1d0 0x7f19881a5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f1987400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1988109810 0x7f198819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ca00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f198810a1d0 0x7f19881a5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60650/0 (socket says 172.21.15.16:60650) 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f198ca00640 1 -- 172.21.15.16:0/1396673382 learned_addr learned my addr 172.21.15.16:0/1396673382 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.963+0000 7f1987e00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1988108620 0x7f198819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1987400640 1 -- 172.21.15.16:0/1396673382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1988108620 msgr2=0x7f198819e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1987400640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1988108620 0x7f198819e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1987400640 1 -- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f198810a1d0 msgr2=0x7f19881a5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1987400640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f198810a1d0 0x7f19881a5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:08.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1987400640 1 -- 172.21.15.16:0/1396673382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19881a6530 con 0x7f1988109810 2024-04-13T03:32:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1987e00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1988108620 0x7f198819e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:32:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f198ca00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f198810a1d0 0x7f19881a5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1987400640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1988109810 0x7f198819ee50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1974036040 tx=0x7f197402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1984c00640 1 -- 172.21.15.16:0/1396673382 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974043020 con 0x7f1988109810 2024-04-13T03:32:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1984c00640 1 -- 172.21.15.16:0/1396673382 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19740054e0 con 0x7f1988109810 2024-04-13T03:32:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f1984c00640 1 -- 172.21.15.16:0/1396673382 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974003c30 con 0x7f1988109810 2024-04-13T03:32:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.964+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19881a67c0 con 0x7f1988109810 2024-04-13T03:32:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.965+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19881a6c90 con 0x7f1988109810 2024-04-13T03:32:08.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.966+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1988069230 con 0x7f1988109810 2024-04-13T03:32:08.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.967+0000 7f1984c00640 1 -- 172.21.15.16:0/1396673382 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1974046020 con 0x7f1988109810 2024-04-13T03:32:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.969+0000 7f1984c00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f196c0a3c40 0x7f196c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.969+0000 7f1984c00640 1 -- 172.21.15.16:0/1396673382 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f19740f2750 con 0x7f1988109810 2024-04-13T03:32:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.969+0000 7f1987e00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f196c0a3c40 0x7f196c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:08.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.970+0000 7f1987e00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f196c0a3c40 0x7f196c0a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1988109670 tx=0x7f1978005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:08.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:08.973+0000 7f1984c00640 1 -- 172.21.15.16:0/1396673382 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f197404d020 con 0x7f1988109810 2024-04-13T03:32:09.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.226+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f198802d5e0 con 0x7f1988109810 2024-04-13T03:32:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.228+0000 7f1984c00640 1 -- 172.21.15.16:0/1396673382 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11934 (secure 0 0 0) 0x7f19740bee40 con 0x7f1988109810 2024-04-13T03:32:09.229 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:32:09.229 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:32:07.796132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24530,"mds_2":24523,"mds_3":24505,"mds_4":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24505,24500,24587,24523,24530]},"id":1}]} 2024-04-13T03:32:09.229 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 26 2024-04-13T03:32:09.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.230+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f196c0a3c40 msgr2=0x7f196c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.230+0000 7f198ea00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f196c0a3c40 0x7f196c0a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1988109670 tx=0x7f1978005f90 comp rx=0 tx=0).stop 2024-04-13T03:32:09.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.230+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1988109810 msgr2=0x7f198819ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.230+0000 7f198ea00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1988109810 0x7f198819ee50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1974036040 tx=0x7f197402c650 comp rx=0 tx=0).stop 2024-04-13T03:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.231+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 shutdown_connections 2024-04-13T03:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.231+0000 7f198ea00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f196c0a3c40 0x7f196c0a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.231+0000 7f198ea00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f198810a1d0 0x7f19881a5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.231+0000 7f198ea00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1988109810 0x7f198819ee50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.231+0000 7f198ea00640 1 --2- 172.21.15.16:0/1396673382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1988108620 0x7f198819e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.231+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 >> 172.21.15.16:0/1396673382 conn(0x7f19880775e0 msgr2=0x7f198810cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.231+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 shutdown_connections 2024-04-13T03:32:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.231+0000 7f198ea00640 1 -- 172.21.15.16:0/1396673382 wait complete. 2024-04-13T03:32:09.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:32:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:09 smithi119 ceph-mon[35835]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:09 smithi119 ceph-mon[35835]: pgmap v334: 129 pgs: 129 active+clean; 584 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 6 op/s 2024-04-13T03:32:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:09 smithi119 ceph-mon[35835]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:09 smithi119 ceph-mon[35835]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-13T03:32:09.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:09 smithi016 ceph-mon[31772]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:09.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:09 smithi016 ceph-mon[31772]: pgmap v334: 129 pgs: 129 active+clean; 584 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 6 op/s 2024-04-13T03:32:09.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:09 smithi016 ceph-mon[31772]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:09.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:09 smithi016 ceph-mon[31772]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-13T03:32:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:09 smithi188 ceph-mon[36398]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:09 smithi188 ceph-mon[36398]: pgmap v334: 129 pgs: 129 active+clean; 584 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 6 op/s 2024-04-13T03:32:09.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:09 smithi188 ceph-mon[36398]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:09.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:09 smithi188 ceph-mon[36398]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-13T03:32:09.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.476+0000 7f7d87200640 1 -- 172.21.15.16:0/500864482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 msgr2=0x7f7d8810c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.476+0000 7f7d87200640 1 --2- 172.21.15.16:0/500864482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 0x7f7d8810c480 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d80009900 tx=0x7f7d8002bc40 comp rx=0 tx=0).stop 2024-04-13T03:32:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.476+0000 7f7d87200640 1 -- 172.21.15.16:0/500864482 shutdown_connections 2024-04-13T03:32:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.476+0000 7f7d87200640 1 --2- 172.21.15.16:0/500864482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 0x7f7d8810c480 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.476+0000 7f7d87200640 1 --2- 172.21.15.16:0/500864482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d88105230 0x7f7d881056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.476+0000 7f7d87200640 1 --2- 172.21.15.16:0/500864482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d88104040 0x7f7d88104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.476+0000 7f7d87200640 1 -- 172.21.15.16:0/500864482 >> 172.21.15.16:0/500864482 conn(0x7f7d880ff5d0 msgr2=0x7f7d881019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.476+0000 7f7d87200640 1 -- 172.21.15.16:0/500864482 shutdown_connections 2024-04-13T03:32:09.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.476+0000 7f7d87200640 1 -- 172.21.15.16:0/500864482 wait complete. 2024-04-13T03:32:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.477+0000 7f7d87200640 1 Processor -- start 2024-04-13T03:32:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.477+0000 7f7d87200640 1 -- start start 2024-04-13T03:32:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d87200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d88104040 0x7f7d8819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d87200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d88105230 0x7f7d8819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d87200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 0x7f7d881a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d87200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8810fce0 con 0x7f7d88105bf0 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d87200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8810fb60 con 0x7f7d88104040 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d87200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8810fe60 con 0x7f7d88105230 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d86800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 0x7f7d881a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d85e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d88104040 0x7f7d8819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d86800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 0x7f7d881a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60662/0 (socket says 172.21.15.16:60662) 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d86800640 1 -- 172.21.15.16:0/1544435700 learned_addr learned my addr 172.21.15.16:0/1544435700 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.478+0000 7f7d7d400640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d88105230 0x7f7d8819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d86800640 1 -- 172.21.15.16:0/1544435700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d88105230 msgr2=0x7f7d8819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d86800640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d88105230 0x7f7d8819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d86800640 1 -- 172.21.15.16:0/1544435700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d88104040 msgr2=0x7f7d8819a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d86800640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d88104040 0x7f7d8819a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d86800640 1 -- 172.21.15.16:0/1544435700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d881a1ee0 con 0x7f7d88105bf0 2024-04-13T03:32:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d86800640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 0x7f7d881a1870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7d8002c180 tx=0x7f7d80005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d7f400640 1 -- 172.21.15.16:0/1544435700 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80041070 con 0x7f7d88105bf0 2024-04-13T03:32:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d7f400640 1 -- 172.21.15.16:0/1544435700 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d800054e0 con 0x7f7d88105bf0 2024-04-13T03:32:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d7f400640 1 -- 172.21.15.16:0/1544435700 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80034440 con 0x7f7d88105bf0 2024-04-13T03:32:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.479+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d881a2170 con 0x7f7d88105bf0 2024-04-13T03:32:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.480+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d881a2610 con 0x7f7d88105bf0 2024-04-13T03:32:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.481+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d8810b300 con 0x7f7d88105bf0 2024-04-13T03:32:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.483+0000 7f7d7f400640 1 -- 172.21.15.16:0/1544435700 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7d80044070 con 0x7f7d88105bf0 2024-04-13T03:32:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.484+0000 7f7d7f400640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7d6c0a3cc0 0x7f7d6c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.484+0000 7f7d7f400640 1 -- 172.21.15.16:0/1544435700 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f7d800f2310 con 0x7f7d88105bf0 2024-04-13T03:32:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.484+0000 7f7d85e00640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7d6c0a3cc0 0x7f7d6c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.485+0000 7f7d85e00640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7d6c0a3cc0 0x7f7d6c0a6180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7d780099a0 tx=0x7f7d78009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.489+0000 7f7d7f400640 1 -- 172.21.15.16:0/1544435700 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d800be410 con 0x7f7d88105bf0 2024-04-13T03:32:09.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.706+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d88062bc0 con 0x7f7d88105bf0 2024-04-13T03:32:09.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.709+0000 7f7d7f400640 1 -- 172.21.15.16:0/1544435700 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7f7d800c2710 con 0x7f7d88105bf0 2024-04-13T03:32:09.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:32:09.711 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:32:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7d6c0a3cc0 msgr2=0x7f7d6c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7d6c0a3cc0 0x7f7d6c0a6180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7d780099a0 tx=0x7f7d78009500 comp rx=0 tx=0).stop 2024-04-13T03:32:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 msgr2=0x7f7d881a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 0x7f7d881a1870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7d8002c180 tx=0x7f7d80005d50 comp rx=0 tx=0).stop 2024-04-13T03:32:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 shutdown_connections 2024-04-13T03:32:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7d6c0a3cc0 0x7f7d6c0a6180 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d88105bf0 0x7f7d881a1870 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d88105230 0x7f7d8819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 --2- 172.21.15.16:0/1544435700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d88104040 0x7f7d8819a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.712+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 >> 172.21.15.16:0/1544435700 conn(0x7f7d880ff5d0 msgr2=0x7f7d881084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.713+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 shutdown_connections 2024-04-13T03:32:09.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.713+0000 7f7d87200640 1 -- 172.21.15.16:0/1544435700 wait complete. 2024-04-13T03:32:09.735 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-13T03:32:09.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-13T03:32:09.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.954+0000 7fb6f6000640 1 -- 172.21.15.16:0/3833719772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 msgr2=0x7fb6f0103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.954+0000 7fb6f6000640 1 --2- 172.21.15.16:0/3833719772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 0x7fb6f0103430 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8009900 tx=0x7fb6d802bc30 comp rx=0 tx=0).stop 2024-04-13T03:32:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.954+0000 7fb6f6000640 1 -- 172.21.15.16:0/3833719772 shutdown_connections 2024-04-13T03:32:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.954+0000 7fb6f6000640 1 --2- 172.21.15.16:0/3833719772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6f0103970 0x7fb6f010a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.954+0000 7fb6f6000640 1 --2- 172.21.15.16:0/3833719772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 0x7fb6f0103430 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.954+0000 7fb6f6000640 1 --2- 172.21.15.16:0/3833719772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6f0101e60 0x7fb6f0102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.954+0000 7fb6f6000640 1 -- 172.21.15.16:0/3833719772 >> 172.21.15.16:0/3833719772 conn(0x7fb6f00fd3d0 msgr2=0x7fb6f00ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.954+0000 7fb6f6000640 1 -- 172.21.15.16:0/3833719772 shutdown_connections 2024-04-13T03:32:09.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.954+0000 7fb6f6000640 1 -- 172.21.15.16:0/3833719772 wait complete. 2024-04-13T03:32:09.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.955+0000 7fb6f6000640 1 Processor -- start 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.955+0000 7fb6f6000640 1 -- start start 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.955+0000 7fb6f6000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6f0101e60 0x7fb6f0075d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6f6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 0x7fb6f00743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6f6000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6f0103970 0x7fb6f00748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6f6000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f010dae0 con 0x7fb6f0102fb0 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6f6000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f010d960 con 0x7fb6f0101e60 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6f6000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f010dc60 con 0x7fb6f0103970 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 0x7fb6f00743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6efe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 0x7fb6f00743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60674/0 (socket says 172.21.15.16:60674) 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6efe00640 1 -- 172.21.15.16:0/4233712703 learned_addr learned my addr 172.21.15.16:0/4233712703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:09.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6f4c00640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6f0101e60 0x7fb6f0075d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6f5600640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6f0103970 0x7fb6f00748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6efe00640 1 -- 172.21.15.16:0/4233712703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6f0103970 msgr2=0x7fb6f00748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6efe00640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6f0103970 0x7fb6f00748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6efe00640 1 -- 172.21.15.16:0/4233712703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6f0101e60 msgr2=0x7fb6f0075d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6efe00640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6f0101e60 0x7fb6f0075d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.956+0000 7fb6efe00640 1 -- 172.21.15.16:0/4233712703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f0075180 con 0x7fb6f0102fb0 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.957+0000 7fb6efe00640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 0x7fb6f00743b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8000c00 tx=0x7fb6d8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.957+0000 7fb6ed600640 1 -- 172.21.15.16:0/4233712703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8041070 con 0x7fb6f0102fb0 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.957+0000 7fb6ed600640 1 -- 172.21.15.16:0/4233712703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6d80044d0 con 0x7fb6f0102fb0 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.957+0000 7fb6ed600640 1 -- 172.21.15.16:0/4233712703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8034440 con 0x7fb6f0102fb0 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.957+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f01a2260 con 0x7fb6f0102fb0 2024-04-13T03:32:09.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.957+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6f01a27b0 con 0x7fb6f0102fb0 2024-04-13T03:32:09.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.958+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6f0109240 con 0x7fb6f0102fb0 2024-04-13T03:32:09.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.962+0000 7fb6ed600640 1 -- 172.21.15.16:0/4233712703 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb6d8044070 con 0x7fb6f0102fb0 2024-04-13T03:32:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.963+0000 7fb6ed600640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6d40a3c40 0x7fb6d40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.964+0000 7fb6ed600640 1 -- 172.21.15.16:0/4233712703 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb6d80f2430 con 0x7fb6f0102fb0 2024-04-13T03:32:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.964+0000 7fb6f4c00640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6d40a3c40 0x7fb6d40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:09.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.965+0000 7fb6f4c00640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6d40a3c40 0x7fb6d40a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00095f0 tx=0x7fb6e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:09.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:09.966+0000 7fb6ed600640 1 -- 172.21.15.16:0/4233712703 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6d80be4d0 con 0x7fb6f0102fb0 2024-04-13T03:32:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.173+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6f0062400 con 0x7fb6d40a3c40 2024-04-13T03:32:10.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:10 smithi188 ceph-mon[36398]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:10.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:10 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1396673382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:32:10.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:10 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1544435700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:10.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.197+0000 7fb6ed600640 1 -- 172.21.15.16:0/4233712703 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340692 (secure 0 0 0) 0x7fb6f0062400 con 0x7fb6d40a3c40 2024-04-13T03:32:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.204+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6d40a3c40 msgr2=0x7fb6d40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.204+0000 7fb6f6000640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6d40a3c40 0x7fb6d40a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00095f0 tx=0x7fb6e0005f90 comp rx=0 tx=0).stop 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.204+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 msgr2=0x7fb6f00743b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.204+0000 7fb6f6000640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 0x7fb6f00743b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8000c00 tx=0x7fb6d8005d50 comp rx=0 tx=0).stop 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.205+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 shutdown_connections 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.205+0000 7fb6f6000640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6d40a3c40 0x7fb6d40a6100 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.205+0000 7fb6f6000640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6f0103970 0x7fb6f00748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.205+0000 7fb6f6000640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6f0102fb0 0x7fb6f00743b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.205+0000 7fb6f6000640 1 --2- 172.21.15.16:0/4233712703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6f0101e60 0x7fb6f0075d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.205+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 >> 172.21.15.16:0/4233712703 conn(0x7fb6f00fd3d0 msgr2=0x7fb6f0106410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:10.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.205+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 shutdown_connections 2024-04-13T03:32:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.205+0000 7fb6f6000640 1 -- 172.21.15.16:0/4233712703 wait complete. 2024-04-13T03:32:10.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:10 smithi016 ceph-mon[31772]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:10.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:10 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1396673382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:32:10.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:10 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1544435700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:10.415 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-04-13T03:32:09.879263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637703,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":158,"num_read_kb":176,"num_write":358,"num_write_kb":3044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":294,"ondisk_log_size":294,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369448,"kb_used_data":11976,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572376,"statfs":{"total":1151940427776,"available":1151562113024,"internally_reserved":0,"allocated":12263424,"data_stored":7673677,"data_compressed":286806,"data_compressed_allocated":385024,"data_compressed_original":2269184,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44476,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":22,"num_read_kb":57,"num_write":130,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005919"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563991+0000","last_change":"2024-04-13T03:31:37.597340+0000","last_active":"2024-04-13T03:31:43.563991+0000","last_peered":"2024-04-13T03:31:43.563991+0000","last_clean":"2024-04-13T03:31:43.563991+0000","last_became_active":"2024-04-13T03:31:37.597081+0000","last_became_peered":"2024-04-13T03:31:37.597081+0000","last_unstale":"2024-04-13T03:31:43.563991+0000","last_undegraded":"2024-04-13T03:31:43.563991+0000","last_fullsized":"2024-04-13T03:31:43.563991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:10:36.679921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562649+0000","last_change":"2024-04-13T03:31:37.578150+0000","last_active":"2024-04-13T03:31:43.562649+0000","last_peered":"2024-04-13T03:31:43.562649+0000","last_clean":"2024-04-13T03:31:43.562649+0000","last_became_active":"2024-04-13T03:31:37.574207+0000","last_became_peered":"2024-04-13T03:31:37.574207+0000","last_unstale":"2024-04-13T03:31:43.562649+0000","last_undegraded":"2024-04-13T03:31:43.562649+0000","last_fullsized":"2024-04-13T03:31:43.562649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:38:41.999046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562348+0000","last_change":"2024-04-13T03:31:37.581782+0000","last_active":"2024-04-13T03:31:43.562348+0000","last_peered":"2024-04-13T03:31:43.562348+0000","last_clean":"2024-04-13T03:31:43.562348+0000","last_became_active":"2024-04-13T03:31:37.581448+0000","last_became_peered":"2024-04-13T03:31:37.581448+0000","last_unstale":"2024-04-13T03:31:43.562348+0000","last_undegraded":"2024-04-13T03:31:43.562348+0000","last_fullsized":"2024-04-13T03:31:43.562348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:56:44.538333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836121+0000","last_change":"2024-04-13T03:31:37.578982+0000","last_active":"2024-04-13T03:31:42.836121+0000","last_peered":"2024-04-13T03:31:42.836121+0000","last_clean":"2024-04-13T03:31:42.836121+0000","last_became_active":"2024-04-13T03:31:37.578375+0000","last_became_peered":"2024-04-13T03:31:37.578375+0000","last_unstale":"2024-04-13T03:31:42.836121+0000","last_undegraded":"2024-04-13T03:31:42.836121+0000","last_fullsized":"2024-04-13T03:31:42.836121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:13:36.312872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562738+0000","last_change":"2024-04-13T03:31:37.577546+0000","last_active":"2024-04-13T03:31:43.562738+0000","last_peered":"2024-04-13T03:31:43.562738+0000","last_clean":"2024-04-13T03:31:43.562738+0000","last_became_active":"2024-04-13T03:31:37.577238+0000","last_became_peered":"2024-04-13T03:31:37.577238+0000","last_unstale":"2024-04-13T03:31:43.562738+0000","last_undegraded":"2024-04-13T03:31:43.562738+0000","last_fullsized":"2024-04-13T03:31:43.562738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:25:17.759986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562623+0000","last_change":"2024-04-13T03:31:37.585806+0000","last_active":"2024-04-13T03:31:43.562623+0000","last_peered":"2024-04-13T03:31:43.562623+0000","last_clean":"2024-04-13T03:31:43.562623+0000","last_became_active":"2024-04-13T03:31:37.585448+0000","last_became_peered":"2024-04-13T03:31:37.585448+0000","last_unstale":"2024-04-13T03:31:43.562623+0000","last_undegraded":"2024-04-13T03:31:43.562623+0000","last_fullsized":"2024-04-13T03:31:43.562623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:32:19.336091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562233+0000","last_change":"2024-04-13T03:31:37.579546+0000","last_active":"2024-04-13T03:31:43.562233+0000","last_peered":"2024-04-13T03:31:43.562233+0000","last_clean":"2024-04-13T03:31:43.562233+0000","last_became_active":"2024-04-13T03:31:37.579165+0000","last_became_peered":"2024-04-13T03:31:37.579165+0000","last_unstale":"2024-04-13T03:31:43.562233+0000","last_undegraded":"2024-04-13T03:31:43.562233+0000","last_fullsized":"2024-04-13T03:31:43.562233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:55:51.788344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248239+0000","last_change":"2024-04-13T03:31:37.587374+0000","last_active":"2024-04-13T03:31:43.248239+0000","last_peered":"2024-04-13T03:31:43.248239+0000","last_clean":"2024-04-13T03:31:43.248239+0000","last_became_active":"2024-04-13T03:31:37.586998+0000","last_became_peered":"2024-04-13T03:31:37.586998+0000","last_unstale":"2024-04-13T03:31:43.248239+0000","last_undegraded":"2024-04-13T03:31:43.248239+0000","last_fullsized":"2024-04-13T03:31:43.248239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:16:46.441001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422978+0000","last_change":"2024-04-13T03:31:37.580579+0000","last_active":"2024-04-13T03:31:43.422978+0000","last_peered":"2024-04-13T03:31:43.422978+0000","last_clean":"2024-04-13T03:31:43.422978+0000","last_became_active":"2024-04-13T03:31:37.580243+0000","last_became_peered":"2024-04-13T03:31:37.580243+0000","last_unstale":"2024-04-13T03:31:43.422978+0000","last_undegraded":"2024-04-13T03:31:43.422978+0000","last_fullsized":"2024-04-13T03:31:43.422978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:42.080260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562737+0000","last_change":"2024-04-13T03:31:37.567731+0000","last_active":"2024-04-13T03:31:43.562737+0000","last_peered":"2024-04-13T03:31:43.562737+0000","last_clean":"2024-04-13T03:31:43.562737+0000","last_became_active":"2024-04-13T03:31:37.567402+0000","last_became_peered":"2024-04-13T03:31:37.567402+0000","last_unstale":"2024-04-13T03:31:43.562737+0000","last_undegraded":"2024-04-13T03:31:43.562737+0000","last_fullsized":"2024-04-13T03:31:43.562737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:54:46.976698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563160+0000","last_change":"2024-04-13T03:31:37.576379+0000","last_active":"2024-04-13T03:31:43.563160+0000","last_peered":"2024-04-13T03:31:43.563160+0000","last_clean":"2024-04-13T03:31:43.563160+0000","last_became_active":"2024-04-13T03:31:37.576062+0000","last_became_peered":"2024-04-13T03:31:37.576062+0000","last_unstale":"2024-04-13T03:31:43.563160+0000","last_undegraded":"2024-04-13T03:31:43.563160+0000","last_fullsized":"2024-04-13T03:31:43.563160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:21:17.507656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835986+0000","last_change":"2024-04-13T03:31:37.578667+0000","last_active":"2024-04-13T03:31:42.835986+0000","last_peered":"2024-04-13T03:31:42.835986+0000","last_clean":"2024-04-13T03:31:42.835986+0000","last_became_active":"2024-04-13T03:31:37.577794+0000","last_became_peered":"2024-04-13T03:31:37.577794+0000","last_unstale":"2024-04-13T03:31:42.835986+0000","last_undegraded":"2024-04-13T03:31:42.835986+0000","last_fullsized":"2024-04-13T03:31:42.835986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:48:51.858309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846709+0000","last_change":"2024-04-13T03:31:37.576827+0000","last_active":"2024-04-13T03:31:42.846709+0000","last_peered":"2024-04-13T03:31:42.846709+0000","last_clean":"2024-04-13T03:31:42.846709+0000","last_became_active":"2024-04-13T03:31:37.576418+0000","last_became_peered":"2024-04-13T03:31:37.576418+0000","last_unstale":"2024-04-13T03:31:42.846709+0000","last_undegraded":"2024-04-13T03:31:42.846709+0000","last_fullsized":"2024-04-13T03:31:42.846709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:45:22.462714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562207+0000","last_change":"2024-04-13T03:31:37.577623+0000","last_active":"2024-04-13T03:31:43.562207+0000","last_peered":"2024-04-13T03:31:43.562207+0000","last_clean":"2024-04-13T03:31:43.562207+0000","last_became_active":"2024-04-13T03:31:37.577175+0000","last_became_peered":"2024-04-13T03:31:37.577175+0000","last_unstale":"2024-04-13T03:31:43.562207+0000","last_undegraded":"2024-04-13T03:31:43.562207+0000","last_fullsized":"2024-04-13T03:31:43.562207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:52:29.604492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562093+0000","last_change":"2024-04-13T03:31:37.581036+0000","last_active":"2024-04-13T03:31:43.562093+0000","last_peered":"2024-04-13T03:31:43.562093+0000","last_clean":"2024-04-13T03:31:43.562093+0000","last_became_active":"2024-04-13T03:31:37.580600+0000","last_became_peered":"2024-04-13T03:31:37.580600+0000","last_unstale":"2024-04-13T03:31:43.562093+0000","last_undegraded":"2024-04-13T03:31:43.562093+0000","last_fullsized":"2024-04-13T03:31:43.562093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:12:29.001001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422361+0000","last_change":"2024-04-13T03:31:37.581073+0000","last_active":"2024-04-13T03:31:43.422361+0000","last_peered":"2024-04-13T03:31:43.422361+0000","last_clean":"2024-04-13T03:31:43.422361+0000","last_became_active":"2024-04-13T03:31:37.580719+0000","last_became_peered":"2024-04-13T03:31:37.580719+0000","last_unstale":"2024-04-13T03:31:43.422361+0000","last_undegraded":"2024-04-13T03:31:43.422361+0000","last_fullsized":"2024-04-13T03:31:43.422361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:22.542263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563743+0000","last_change":"2024-04-13T03:31:37.597572+0000","last_active":"2024-04-13T03:31:43.563743+0000","last_peered":"2024-04-13T03:31:43.563743+0000","last_clean":"2024-04-13T03:31:43.563743+0000","last_became_active":"2024-04-13T03:31:37.597244+0000","last_became_peered":"2024-04-13T03:31:37.597244+0000","last_unstale":"2024-04-13T03:31:43.563743+0000","last_undegraded":"2024-04-13T03:31:43.563743+0000","last_fullsized":"2024-04-13T03:31:43.563743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:39:13.365427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835972+0000","last_change":"2024-04-13T03:31:37.596749+0000","last_active":"2024-04-13T03:31:42.835972+0000","last_peered":"2024-04-13T03:31:42.835972+0000","last_clean":"2024-04-13T03:31:42.835972+0000","last_became_active":"2024-04-13T03:31:37.592409+0000","last_became_peered":"2024-04-13T03:31:37.592409+0000","last_unstale":"2024-04-13T03:31:42.835972+0000","last_undegraded":"2024-04-13T03:31:42.835972+0000","last_fullsized":"2024-04-13T03:31:42.835972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:35:22.360224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563035+0000","last_change":"2024-04-13T03:31:37.571354+0000","last_active":"2024-04-13T03:31:43.563035+0000","last_peered":"2024-04-13T03:31:43.563035+0000","last_clean":"2024-04-13T03:31:43.563035+0000","last_became_active":"2024-04-13T03:31:37.571012+0000","last_became_peered":"2024-04-13T03:31:37.571012+0000","last_unstale":"2024-04-13T03:31:43.563035+0000","last_undegraded":"2024-04-13T03:31:43.563035+0000","last_fullsized":"2024-04-13T03:31:43.563035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:50:26.492090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562015+0000","last_change":"2024-04-13T03:31:41.844448+0000","last_active":"2024-04-13T03:31:43.562015+0000","last_peered":"2024-04-13T03:31:43.562015+0000","last_clean":"2024-04-13T03:31:43.562015+0000","last_became_active":"2024-04-13T03:31:41.844143+0000","last_became_peered":"2024-04-13T03:31:41.844143+0000","last_unstale":"2024-04-13T03:31:43.562015+0000","last_undegraded":"2024-04-13T03:31:43.562015+0000","last_fullsized":"2024-04-13T03:31:43.562015+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:26:40.811950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.055705+0000","last_change":"2024-04-13T03:31:36.581372+0000","last_active":"2024-04-13T03:32:05.055705+0000","last_peered":"2024-04-13T03:32:05.055705+0000","last_clean":"2024-04-13T03:32:05.055705+0000","last_became_active":"2024-04-13T03:31:36.580599+0000","last_became_peered":"2024-04-13T03:31:36.580599+0000","last_unstale":"2024-04-13T03:32:05.055705+0000","last_undegraded":"2024-04-13T03:32:05.055705+0000","last_fullsized":"2024-04-13T03:32:05.055705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:10:09.839049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.249836+0000","last_change":"2024-04-13T03:31:37.584511+0000","last_active":"2024-04-13T03:31:43.249836+0000","last_peered":"2024-04-13T03:31:43.249836+0000","last_clean":"2024-04-13T03:31:43.249836+0000","last_became_active":"2024-04-13T03:31:37.584094+0000","last_became_peered":"2024-04-13T03:31:37.584094+0000","last_unstale":"2024-04-13T03:31:43.249836+0000","last_undegraded":"2024-04-13T03:31:43.249836+0000","last_fullsized":"2024-04-13T03:31:43.249836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:47:36.163763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835423+0000","last_change":"2024-04-13T03:31:41.834169+0000","last_active":"2024-04-13T03:31:42.835423+0000","last_peered":"2024-04-13T03:31:42.835423+0000","last_clean":"2024-04-13T03:31:42.835423+0000","last_became_active":"2024-04-13T03:31:41.833545+0000","last_became_peered":"2024-04-13T03:31:41.833545+0000","last_unstale":"2024-04-13T03:31:42.835423+0000","last_undegraded":"2024-04-13T03:31:42.835423+0000","last_fullsized":"2024-04-13T03:31:42.835423+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:37:29.657978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.249994+0000","last_change":"2024-04-13T03:31:36.596225+0000","last_active":"2024-04-13T03:31:43.249994+0000","last_peered":"2024-04-13T03:31:43.249994+0000","last_clean":"2024-04-13T03:31:43.249994+0000","last_became_active":"2024-04-13T03:31:36.595520+0000","last_became_peered":"2024-04-13T03:31:36.595520+0000","last_unstale":"2024-04-13T03:31:43.249994+0000","last_undegraded":"2024-04-13T03:31:43.249994+0000","last_fullsized":"2024-04-13T03:31:43.249994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:15:35.172065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562506+0000","last_change":"2024-04-13T03:31:37.587264+0000","last_active":"2024-04-13T03:31:43.562506+0000","last_peered":"2024-04-13T03:31:43.562506+0000","last_clean":"2024-04-13T03:31:43.562506+0000","last_became_active":"2024-04-13T03:31:37.585574+0000","last_became_peered":"2024-04-13T03:31:37.585574+0000","last_unstale":"2024-04-13T03:31:43.562506+0000","last_undegraded":"2024-04-13T03:31:43.562506+0000","last_fullsized":"2024-04-13T03:31:43.562506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:39.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835558+0000","last_change":"2024-04-13T03:31:41.827237+0000","last_active":"2024-04-13T03:31:42.835558+0000","last_peered":"2024-04-13T03:31:42.835558+0000","last_clean":"2024-04-13T03:31:42.835558+0000","last_became_active":"2024-04-13T03:31:41.826799+0000","last_became_peered":"2024-04-13T03:31:41.826799+0000","last_unstale":"2024-04-13T03:31:42.835558+0000","last_undegraded":"2024-04-13T03:31:42.835558+0000","last_fullsized":"2024-04-13T03:31:42.835558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:26:14.337654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.043515+0000","last_change":"2024-04-13T03:31:36.564752+0000","last_active":"2024-04-13T03:32:05.043515+0000","last_peered":"2024-04-13T03:32:05.043515+0000","last_clean":"2024-04-13T03:32:05.043515+0000","last_became_active":"2024-04-13T03:31:36.564413+0000","last_became_peered":"2024-04-13T03:31:36.564413+0000","last_unstale":"2024-04-13T03:32:05.043515+0000","last_undegraded":"2024-04-13T03:32:05.043515+0000","last_fullsized":"2024-04-13T03:32:05.043515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:34:52.369875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422639+0000","last_change":"2024-04-13T03:31:37.579316+0000","last_active":"2024-04-13T03:31:43.422639+0000","last_peered":"2024-04-13T03:31:43.422639+0000","last_clean":"2024-04-13T03:31:43.422639+0000","last_became_active":"2024-04-13T03:31:37.578971+0000","last_became_peered":"2024-04-13T03:31:37.578971+0000","last_unstale":"2024-04-13T03:31:43.422639+0000","last_undegraded":"2024-04-13T03:31:43.422639+0000","last_fullsized":"2024-04-13T03:31:43.422639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:54:57.013722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835551+0000","last_change":"2024-04-13T03:31:41.841120+0000","last_active":"2024-04-13T03:31:42.835551+0000","last_peered":"2024-04-13T03:31:42.835551+0000","last_clean":"2024-04-13T03:31:42.835551+0000","last_became_active":"2024-04-13T03:31:41.840833+0000","last_became_peered":"2024-04-13T03:31:41.840833+0000","last_unstale":"2024-04-13T03:31:42.835551+0000","last_undegraded":"2024-04-13T03:31:42.835551+0000","last_fullsized":"2024-04-13T03:31:42.835551+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:13:09.979055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563354+0000","last_change":"2024-04-13T03:31:36.561428+0000","last_active":"2024-04-13T03:31:43.563354+0000","last_peered":"2024-04-13T03:31:43.563354+0000","last_clean":"2024-04-13T03:31:43.563354+0000","last_became_active":"2024-04-13T03:31:36.561100+0000","last_became_peered":"2024-04-13T03:31:36.561100+0000","last_unstale":"2024-04-13T03:31:43.563354+0000","last_undegraded":"2024-04-13T03:31:43.563354+0000","last_fullsized":"2024-04-13T03:31:43.563354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:04:32.125292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834499+0000","last_change":"2024-04-13T03:31:37.584932+0000","last_active":"2024-04-13T03:31:42.834499+0000","last_peered":"2024-04-13T03:31:42.834499+0000","last_clean":"2024-04-13T03:31:42.834499+0000","last_became_active":"2024-04-13T03:31:37.584607+0000","last_became_peered":"2024-04-13T03:31:37.584607+0000","last_unstale":"2024-04-13T03:31:42.834499+0000","last_undegraded":"2024-04-13T03:31:42.834499+0000","last_fullsized":"2024-04-13T03:31:42.834499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:15:50.573605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562972+0000","last_change":"2024-04-13T03:31:41.824295+0000","last_active":"2024-04-13T03:31:43.562972+0000","last_peered":"2024-04-13T03:31:43.562972+0000","last_clean":"2024-04-13T03:31:43.562972+0000","last_became_active":"2024-04-13T03:31:41.823810+0000","last_became_peered":"2024-04-13T03:31:41.823810+0000","last_unstale":"2024-04-13T03:31:43.562972+0000","last_undegraded":"2024-04-13T03:31:43.562972+0000","last_fullsized":"2024-04-13T03:31:43.562972+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:40:08.888519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.814594+0000","last_change":"2024-04-13T03:31:36.558866+0000","last_active":"2024-04-13T03:32:07.814594+0000","last_peered":"2024-04-13T03:32:07.814594+0000","last_clean":"2024-04-13T03:32:07.814594+0000","last_became_active":"2024-04-13T03:31:36.558541+0000","last_became_peered":"2024-04-13T03:31:36.558541+0000","last_unstale":"2024-04-13T03:32:07.814594+0000","last_undegraded":"2024-04-13T03:32:07.814594+0000","last_fullsized":"2024-04-13T03:32:07.814594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:59:29.431084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563115+0000","last_change":"2024-04-13T03:31:37.581298+0000","last_active":"2024-04-13T03:31:43.563115+0000","last_peered":"2024-04-13T03:31:43.563115+0000","last_clean":"2024-04-13T03:31:43.563115+0000","last_became_active":"2024-04-13T03:31:37.580963+0000","last_became_peered":"2024-04-13T03:31:37.580963+0000","last_unstale":"2024-04-13T03:31:43.563115+0000","last_undegraded":"2024-04-13T03:31:43.563115+0000","last_fullsized":"2024-04-13T03:31:43.563115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:47:27.921928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563122+0000","last_change":"2024-04-13T03:31:41.831221+0000","last_active":"2024-04-13T03:31:43.563122+0000","last_peered":"2024-04-13T03:31:43.563122+0000","last_clean":"2024-04-13T03:31:43.563122+0000","last_became_active":"2024-04-13T03:31:41.830246+0000","last_became_peered":"2024-04-13T03:31:41.830246+0000","last_unstale":"2024-04-13T03:31:43.563122+0000","last_undegraded":"2024-04-13T03:31:43.563122+0000","last_fullsized":"2024-04-13T03:31:43.563122+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:08:16.624658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.074395+0000","last_change":"2024-04-13T03:31:36.585534+0000","last_active":"2024-04-13T03:32:06.074395+0000","last_peered":"2024-04-13T03:32:06.074395+0000","last_clean":"2024-04-13T03:32:06.074395+0000","last_became_active":"2024-04-13T03:31:36.585324+0000","last_became_peered":"2024-04-13T03:31:36.585324+0000","last_unstale":"2024-04-13T03:32:06.074395+0000","last_undegraded":"2024-04-13T03:32:06.074395+0000","last_fullsized":"2024-04-13T03:32:06.074395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:12:35.971994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834461+0000","last_change":"2024-04-13T03:31:37.587440+0000","last_active":"2024-04-13T03:31:42.834461+0000","last_peered":"2024-04-13T03:31:42.834461+0000","last_clean":"2024-04-13T03:31:42.834461+0000","last_became_active":"2024-04-13T03:31:37.581413+0000","last_became_peered":"2024-04-13T03:31:37.581413+0000","last_unstale":"2024-04-13T03:31:42.834461+0000","last_undegraded":"2024-04-13T03:31:42.834461+0000","last_fullsized":"2024-04-13T03:31:42.834461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:14:26.066107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563930+0000","last_change":"2024-04-13T03:31:41.848451+0000","last_active":"2024-04-13T03:31:43.563930+0000","last_peered":"2024-04-13T03:31:43.563930+0000","last_clean":"2024-04-13T03:31:43.563930+0000","last_became_active":"2024-04-13T03:31:41.848019+0000","last_became_peered":"2024-04-13T03:31:41.848019+0000","last_unstale":"2024-04-13T03:31:43.563930+0000","last_undegraded":"2024-04-13T03:31:43.563930+0000","last_fullsized":"2024-04-13T03:31:43.563930+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:26:34.558142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.055541+0000","last_change":"2024-04-13T03:31:36.588393+0000","last_active":"2024-04-13T03:32:05.055541+0000","last_peered":"2024-04-13T03:32:05.055541+0000","last_clean":"2024-04-13T03:32:05.055541+0000","last_became_active":"2024-04-13T03:31:36.587184+0000","last_became_peered":"2024-04-13T03:31:36.587184+0000","last_unstale":"2024-04-13T03:32:05.055541+0000","last_undegraded":"2024-04-13T03:32:05.055541+0000","last_fullsized":"2024-04-13T03:32:05.055541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:44:24.985634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248478+0000","last_change":"2024-04-13T03:31:37.600417+0000","last_active":"2024-04-13T03:31:43.248478+0000","last_peered":"2024-04-13T03:31:43.248478+0000","last_clean":"2024-04-13T03:31:43.248478+0000","last_became_active":"2024-04-13T03:31:37.599889+0000","last_became_peered":"2024-04-13T03:31:37.599889+0000","last_unstale":"2024-04-13T03:31:43.248478+0000","last_undegraded":"2024-04-13T03:31:43.248478+0000","last_fullsized":"2024-04-13T03:31:43.248478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:19:13.039127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562624+0000","last_change":"2024-04-13T03:31:41.848432+0000","last_active":"2024-04-13T03:31:43.562624+0000","last_peered":"2024-04-13T03:31:43.562624+0000","last_clean":"2024-04-13T03:31:43.562624+0000","last_became_active":"2024-04-13T03:31:41.843515+0000","last_became_peered":"2024-04-13T03:31:41.843515+0000","last_unstale":"2024-04-13T03:31:43.562624+0000","last_undegraded":"2024-04-13T03:31:43.562624+0000","last_fullsized":"2024-04-13T03:31:43.562624+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:31:15.512541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.078862+0000","last_change":"2024-04-13T03:31:36.560090+0000","last_active":"2024-04-13T03:32:07.078862+0000","last_peered":"2024-04-13T03:32:07.078862+0000","last_clean":"2024-04-13T03:32:07.078862+0000","last_became_active":"2024-04-13T03:31:36.559770+0000","last_became_peered":"2024-04-13T03:31:36.559770+0000","last_unstale":"2024-04-13T03:32:07.078862+0000","last_undegraded":"2024-04-13T03:32:07.078862+0000","last_fullsized":"2024-04-13T03:32:07.078862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:45:51.715456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835246+0000","last_change":"2024-04-13T03:31:37.593845+0000","last_active":"2024-04-13T03:31:42.835246+0000","last_peered":"2024-04-13T03:31:42.835246+0000","last_clean":"2024-04-13T03:31:42.835246+0000","last_became_active":"2024-04-13T03:31:37.593484+0000","last_became_peered":"2024-04-13T03:31:37.593484+0000","last_unstale":"2024-04-13T03:31:42.835246+0000","last_undegraded":"2024-04-13T03:31:42.835246+0000","last_fullsized":"2024-04-13T03:31:42.835246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:38:05.817746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248747+0000","last_change":"2024-04-13T03:31:41.830949+0000","last_active":"2024-04-13T03:31:43.248747+0000","last_peered":"2024-04-13T03:31:43.248747+0000","last_clean":"2024-04-13T03:31:43.248747+0000","last_became_active":"2024-04-13T03:31:41.830540+0000","last_became_peered":"2024-04-13T03:31:41.830540+0000","last_unstale":"2024-04-13T03:31:43.248747+0000","last_undegraded":"2024-04-13T03:31:43.248747+0000","last_fullsized":"2024-04-13T03:31:43.248747+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:09:00.006158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:08.809274+0000","last_change":"2024-04-13T03:31:36.574773+0000","last_active":"2024-04-13T03:32:08.809274+0000","last_peered":"2024-04-13T03:32:08.809274+0000","last_clean":"2024-04-13T03:32:08.809274+0000","last_became_active":"2024-04-13T03:31:36.574511+0000","last_became_peered":"2024-04-13T03:31:36.574511+0000","last_unstale":"2024-04-13T03:32:08.809274+0000","last_undegraded":"2024-04-13T03:32:08.809274+0000","last_fullsized":"2024-04-13T03:32:08.809274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:26:41.010269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422894+0000","last_change":"2024-04-13T03:31:37.583383+0000","last_active":"2024-04-13T03:31:43.422894+0000","last_peered":"2024-04-13T03:31:43.422894+0000","last_clean":"2024-04-13T03:31:43.422894+0000","last_became_active":"2024-04-13T03:31:37.583026+0000","last_became_peered":"2024-04-13T03:31:37.583026+0000","last_unstale":"2024-04-13T03:31:43.422894+0000","last_undegraded":"2024-04-13T03:31:43.422894+0000","last_fullsized":"2024-04-13T03:31:43.422894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:34.790848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835409+0000","last_change":"2024-04-13T03:31:41.845867+0000","last_active":"2024-04-13T03:31:42.835409+0000","last_peered":"2024-04-13T03:31:42.835409+0000","last_clean":"2024-04-13T03:31:42.835409+0000","last_became_active":"2024-04-13T03:31:41.845408+0000","last_became_peered":"2024-04-13T03:31:41.845408+0000","last_unstale":"2024-04-13T03:31:42.835409+0000","last_undegraded":"2024-04-13T03:31:42.835409+0000","last_fullsized":"2024-04-13T03:31:42.835409+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:10:42.830683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:04.034415+0000","last_change":"2024-04-13T03:31:36.562534+0000","last_active":"2024-04-13T03:32:04.034415+0000","last_peered":"2024-04-13T03:32:04.034415+0000","last_clean":"2024-04-13T03:32:04.034415+0000","last_became_active":"2024-04-13T03:31:36.562189+0000","last_became_peered":"2024-04-13T03:31:36.562189+0000","last_unstale":"2024-04-13T03:32:04.034415+0000","last_undegraded":"2024-04-13T03:32:04.034415+0000","last_fullsized":"2024-04-13T03:32:04.034415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:41:27.504982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562634+0000","last_change":"2024-04-13T03:31:37.588637+0000","last_active":"2024-04-13T03:31:43.562634+0000","last_peered":"2024-04-13T03:31:43.562634+0000","last_clean":"2024-04-13T03:31:43.562634+0000","last_became_active":"2024-04-13T03:31:37.583263+0000","last_became_peered":"2024-04-13T03:31:37.583263+0000","last_unstale":"2024-04-13T03:31:43.562634+0000","last_undegraded":"2024-04-13T03:31:43.562634+0000","last_fullsized":"2024-04-13T03:31:43.562634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:07:40.503310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835767+0000","last_change":"2024-04-13T03:31:41.827302+0000","last_active":"2024-04-13T03:31:42.835767+0000","last_peered":"2024-04-13T03:31:42.835767+0000","last_clean":"2024-04-13T03:31:42.835767+0000","last_became_active":"2024-04-13T03:31:41.826822+0000","last_became_peered":"2024-04-13T03:31:41.826822+0000","last_unstale":"2024-04-13T03:31:42.835767+0000","last_undegraded":"2024-04-13T03:31:42.835767+0000","last_fullsized":"2024-04-13T03:31:42.835767+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:00:16.589319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.072654+0000","last_change":"2024-04-13T03:31:36.582710+0000","last_active":"2024-04-13T03:32:06.072654+0000","last_peered":"2024-04-13T03:32:06.072654+0000","last_clean":"2024-04-13T03:32:06.072654+0000","last_became_active":"2024-04-13T03:31:36.582372+0000","last_became_peered":"2024-04-13T03:31:36.582372+0000","last_unstale":"2024-04-13T03:32:06.072654+0000","last_undegraded":"2024-04-13T03:32:06.072654+0000","last_fullsized":"2024-04-13T03:32:06.072654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:38:58.990256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422774+0000","last_change":"2024-04-13T03:31:37.578345+0000","last_active":"2024-04-13T03:31:43.422774+0000","last_peered":"2024-04-13T03:31:43.422774+0000","last_clean":"2024-04-13T03:31:43.422774+0000","last_became_active":"2024-04-13T03:31:37.578026+0000","last_became_peered":"2024-04-13T03:31:37.578026+0000","last_unstale":"2024-04-13T03:31:43.422774+0000","last_undegraded":"2024-04-13T03:31:43.422774+0000","last_fullsized":"2024-04-13T03:31:43.422774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:59:02.080243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835514+0000","last_change":"2024-04-13T03:31:41.829108+0000","last_active":"2024-04-13T03:31:42.835514+0000","last_peered":"2024-04-13T03:31:42.835514+0000","last_clean":"2024-04-13T03:31:42.835514+0000","last_became_active":"2024-04-13T03:31:41.828603+0000","last_became_peered":"2024-04-13T03:31:41.828603+0000","last_unstale":"2024-04-13T03:31:42.835514+0000","last_undegraded":"2024-04-13T03:31:42.835514+0000","last_fullsized":"2024-04-13T03:31:42.835514+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:06:32.753928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'26","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:08.814345+0000","last_change":"2024-04-13T03:31:36.562447+0000","last_active":"2024-04-13T03:32:08.814345+0000","last_peered":"2024-04-13T03:32:08.814345+0000","last_clean":"2024-04-13T03:32:08.814345+0000","last_became_active":"2024-04-13T03:31:36.562094+0000","last_became_peered":"2024-04-13T03:31:36.562094+0000","last_unstale":"2024-04-13T03:32:08.814345+0000","last_undegraded":"2024-04-13T03:32:08.814345+0000","last_fullsized":"2024-04-13T03:32:08.814345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:04:26.847519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41906,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":44,"num_write":28,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563024+0000","last_change":"2024-04-13T03:31:37.580515+0000","last_active":"2024-04-13T03:31:43.563024+0000","last_peered":"2024-04-13T03:31:43.563024+0000","last_clean":"2024-04-13T03:31:43.563024+0000","last_became_active":"2024-04-13T03:31:37.580157+0000","last_became_peered":"2024-04-13T03:31:37.580157+0000","last_unstale":"2024-04-13T03:31:43.563024+0000","last_undegraded":"2024-04-13T03:31:43.563024+0000","last_fullsized":"2024-04-13T03:31:43.563024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:17:17.843273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834856+0000","last_change":"2024-04-13T03:31:41.833555+0000","last_active":"2024-04-13T03:31:42.834856+0000","last_peered":"2024-04-13T03:31:42.834856+0000","last_clean":"2024-04-13T03:31:42.834856+0000","last_became_active":"2024-04-13T03:31:41.833134+0000","last_became_peered":"2024-04-13T03:31:41.833134+0000","last_unstale":"2024-04-13T03:31:42.834856+0000","last_undegraded":"2024-04-13T03:31:42.834856+0000","last_fullsized":"2024-04-13T03:31:42.834856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:22:58.311184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.048771+0000","last_change":"2024-04-13T03:31:36.591127+0000","last_active":"2024-04-13T03:32:05.048771+0000","last_peered":"2024-04-13T03:32:05.048771+0000","last_clean":"2024-04-13T03:32:05.048771+0000","last_became_active":"2024-04-13T03:31:36.590819+0000","last_became_peered":"2024-04-13T03:31:36.590819+0000","last_unstale":"2024-04-13T03:32:05.048771+0000","last_undegraded":"2024-04-13T03:32:05.048771+0000","last_fullsized":"2024-04-13T03:32:05.048771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:54:51.637340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846695+0000","last_change":"2024-04-13T03:31:37.583147+0000","last_active":"2024-04-13T03:31:42.846695+0000","last_peered":"2024-04-13T03:31:42.846695+0000","last_clean":"2024-04-13T03:31:42.846695+0000","last_became_active":"2024-04-13T03:31:37.582781+0000","last_became_peered":"2024-04-13T03:31:37.582781+0000","last_unstale":"2024-04-13T03:31:42.846695+0000","last_undegraded":"2024-04-13T03:31:42.846695+0000","last_fullsized":"2024-04-13T03:31:42.846695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:47:36.879512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563488+0000","last_change":"2024-04-13T03:31:41.833559+0000","last_active":"2024-04-13T03:31:43.563488+0000","last_peered":"2024-04-13T03:31:43.563488+0000","last_clean":"2024-04-13T03:31:43.563488+0000","last_became_active":"2024-04-13T03:31:41.833067+0000","last_became_peered":"2024-04-13T03:31:41.833067+0000","last_unstale":"2024-04-13T03:31:43.563488+0000","last_undegraded":"2024-04-13T03:31:43.563488+0000","last_fullsized":"2024-04-13T03:31:43.563488+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:56:09.027293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.084769+0000","last_change":"2024-04-13T03:31:36.597136+0000","last_active":"2024-04-13T03:32:07.084769+0000","last_peered":"2024-04-13T03:32:07.084769+0000","last_clean":"2024-04-13T03:32:07.084769+0000","last_became_active":"2024-04-13T03:31:36.596789+0000","last_became_peered":"2024-04-13T03:31:36.596789+0000","last_unstale":"2024-04-13T03:32:07.084769+0000","last_undegraded":"2024-04-13T03:32:07.084769+0000","last_fullsized":"2024-04-13T03:32:07.084769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:39:28.371669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562806+0000","last_change":"2024-04-13T03:31:37.584424+0000","last_active":"2024-04-13T03:31:43.562806+0000","last_peered":"2024-04-13T03:31:43.562806+0000","last_clean":"2024-04-13T03:31:43.562806+0000","last_became_active":"2024-04-13T03:31:37.584125+0000","last_became_peered":"2024-04-13T03:31:37.584125+0000","last_unstale":"2024-04-13T03:31:43.562806+0000","last_undegraded":"2024-04-13T03:31:43.562806+0000","last_fullsized":"2024-04-13T03:31:43.562806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:07:53.625486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836253+0000","last_change":"2024-04-13T03:31:41.852755+0000","last_active":"2024-04-13T03:31:42.836253+0000","last_peered":"2024-04-13T03:31:42.836253+0000","last_clean":"2024-04-13T03:31:42.836253+0000","last_became_active":"2024-04-13T03:31:41.851975+0000","last_became_peered":"2024-04-13T03:31:41.851975+0000","last_unstale":"2024-04-13T03:31:42.836253+0000","last_undegraded":"2024-04-13T03:31:42.836253+0000","last_fullsized":"2024-04-13T03:31:42.836253+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:19:40.982525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.064509+0000","last_change":"2024-04-13T03:31:36.559351+0000","last_active":"2024-04-13T03:32:06.064509+0000","last_peered":"2024-04-13T03:32:06.064509+0000","last_clean":"2024-04-13T03:32:06.064509+0000","last_became_active":"2024-04-13T03:31:36.559011+0000","last_became_peered":"2024-04-13T03:31:36.559011+0000","last_unstale":"2024-04-13T03:32:06.064509+0000","last_undegraded":"2024-04-13T03:32:06.064509+0000","last_fullsized":"2024-04-13T03:32:06.064509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:08:08.983088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562255+0000","last_change":"2024-04-13T03:31:37.580884+0000","last_active":"2024-04-13T03:31:43.562255+0000","last_peered":"2024-04-13T03:31:43.562255+0000","last_clean":"2024-04-13T03:31:43.562255+0000","last_became_active":"2024-04-13T03:31:37.580568+0000","last_became_peered":"2024-04-13T03:31:37.580568+0000","last_unstale":"2024-04-13T03:31:43.562255+0000","last_undegraded":"2024-04-13T03:31:43.562255+0000","last_fullsized":"2024-04-13T03:31:43.562255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:42:41.484614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.847286+0000","last_change":"2024-04-13T03:31:41.840912+0000","last_active":"2024-04-13T03:31:42.847286+0000","last_peered":"2024-04-13T03:31:42.847286+0000","last_clean":"2024-04-13T03:31:42.847286+0000","last_became_active":"2024-04-13T03:31:41.840129+0000","last_became_peered":"2024-04-13T03:31:41.840129+0000","last_unstale":"2024-04-13T03:31:42.847286+0000","last_undegraded":"2024-04-13T03:31:42.847286+0000","last_fullsized":"2024-04-13T03:31:42.847286+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:45:33.716345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.046113+0000","last_change":"2024-04-13T03:31:36.580230+0000","last_active":"2024-04-13T03:32:05.046113+0000","last_peered":"2024-04-13T03:32:05.046113+0000","last_clean":"2024-04-13T03:32:05.046113+0000","last_became_active":"2024-04-13T03:31:36.579874+0000","last_became_peered":"2024-04-13T03:31:36.579874+0000","last_unstale":"2024-04-13T03:32:05.046113+0000","last_undegraded":"2024-04-13T03:32:05.046113+0000","last_fullsized":"2024-04-13T03:32:05.046113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:30:17.154748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834014+0000","last_change":"2024-04-13T03:31:37.581937+0000","last_active":"2024-04-13T03:31:42.834014+0000","last_peered":"2024-04-13T03:31:42.834014+0000","last_clean":"2024-04-13T03:31:42.834014+0000","last_became_active":"2024-04-13T03:31:37.581547+0000","last_became_peered":"2024-04-13T03:31:37.581547+0000","last_unstale":"2024-04-13T03:31:42.834014+0000","last_undegraded":"2024-04-13T03:31:42.834014+0000","last_fullsized":"2024-04-13T03:31:42.834014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:19:28.600996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834086+0000","last_change":"2024-04-13T03:31:41.842887+0000","last_active":"2024-04-13T03:31:42.834086+0000","last_peered":"2024-04-13T03:31:42.834086+0000","last_clean":"2024-04-13T03:31:42.834086+0000","last_became_active":"2024-04-13T03:31:41.842537+0000","last_became_peered":"2024-04-13T03:31:41.842537+0000","last_unstale":"2024-04-13T03:31:42.834086+0000","last_undegraded":"2024-04-13T03:31:42.834086+0000","last_fullsized":"2024-04-13T03:31:42.834086+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:44:38.930759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563300+0000","last_change":"2024-04-13T03:31:36.561347+0000","last_active":"2024-04-13T03:31:43.563300+0000","last_peered":"2024-04-13T03:31:43.563300+0000","last_clean":"2024-04-13T03:31:43.563300+0000","last_became_active":"2024-04-13T03:31:36.560969+0000","last_became_peered":"2024-04-13T03:31:36.560969+0000","last_unstale":"2024-04-13T03:31:43.563300+0000","last_undegraded":"2024-04-13T03:31:43.563300+0000","last_fullsized":"2024-04-13T03:31:43.563300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:59:38.825930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563519+0000","last_change":"2024-04-13T03:31:37.587599+0000","last_active":"2024-04-13T03:31:43.563519+0000","last_peered":"2024-04-13T03:31:43.563519+0000","last_clean":"2024-04-13T03:31:43.563519+0000","last_became_active":"2024-04-13T03:31:37.587257+0000","last_became_peered":"2024-04-13T03:31:37.587257+0000","last_unstale":"2024-04-13T03:31:43.563519+0000","last_undegraded":"2024-04-13T03:31:43.563519+0000","last_fullsized":"2024-04-13T03:31:43.563519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:39:57.956174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563296+0000","last_change":"2024-04-13T03:31:41.831134+0000","last_active":"2024-04-13T03:31:43.563296+0000","last_peered":"2024-04-13T03:31:43.563296+0000","last_clean":"2024-04-13T03:31:43.563296+0000","last_became_active":"2024-04-13T03:31:41.830671+0000","last_became_peered":"2024-04-13T03:31:41.830671+0000","last_unstale":"2024-04-13T03:31:43.563296+0000","last_undegraded":"2024-04-13T03:31:43.563296+0000","last_fullsized":"2024-04-13T03:31:43.563296+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:50:59.527951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836188+0000","last_change":"2024-04-13T03:31:36.586008+0000","last_active":"2024-04-13T03:31:42.836188+0000","last_peered":"2024-04-13T03:31:42.836188+0000","last_clean":"2024-04-13T03:31:42.836188+0000","last_became_active":"2024-04-13T03:31:36.583937+0000","last_became_peered":"2024-04-13T03:31:36.583937+0000","last_unstale":"2024-04-13T03:31:42.836188+0000","last_undegraded":"2024-04-13T03:31:42.836188+0000","last_fullsized":"2024-04-13T03:31:42.836188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:17:06.194397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562737+0000","last_change":"2024-04-13T03:31:37.586185+0000","last_active":"2024-04-13T03:31:43.562737+0000","last_peered":"2024-04-13T03:31:43.562737+0000","last_clean":"2024-04-13T03:31:43.562737+0000","last_became_active":"2024-04-13T03:31:37.585509+0000","last_became_peered":"2024-04-13T03:31:37.585509+0000","last_unstale":"2024-04-13T03:31:43.562737+0000","last_undegraded":"2024-04-13T03:31:43.562737+0000","last_fullsized":"2024-04-13T03:31:43.562737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:35:41.064317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846911+0000","last_change":"2024-04-13T03:31:41.847376+0000","last_active":"2024-04-13T03:31:42.846911+0000","last_peered":"2024-04-13T03:31:42.846911+0000","last_clean":"2024-04-13T03:31:42.846911+0000","last_became_active":"2024-04-13T03:31:41.846724+0000","last_became_peered":"2024-04-13T03:31:41.846724+0000","last_unstale":"2024-04-13T03:31:42.846911+0000","last_undegraded":"2024-04-13T03:31:42.846911+0000","last_fullsized":"2024-04-13T03:31:42.846911+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:26:14.046875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562510+0000","last_change":"2024-04-13T03:31:36.582776+0000","last_active":"2024-04-13T03:31:43.562510+0000","last_peered":"2024-04-13T03:31:43.562510+0000","last_clean":"2024-04-13T03:31:43.562510+0000","last_became_active":"2024-04-13T03:31:36.582405+0000","last_became_peered":"2024-04-13T03:31:36.582405+0000","last_unstale":"2024-04-13T03:31:43.562510+0000","last_undegraded":"2024-04-13T03:31:43.562510+0000","last_fullsized":"2024-04-13T03:31:43.562510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:18:56.624442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562925+0000","last_change":"2024-04-13T03:31:37.580552+0000","last_active":"2024-04-13T03:31:43.562925+0000","last_peered":"2024-04-13T03:31:43.562925+0000","last_clean":"2024-04-13T03:31:43.562925+0000","last_became_active":"2024-04-13T03:31:37.576390+0000","last_became_peered":"2024-04-13T03:31:37.576390+0000","last_unstale":"2024-04-13T03:31:43.562925+0000","last_undegraded":"2024-04-13T03:31:43.562925+0000","last_fullsized":"2024-04-13T03:31:43.562925+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:50:48.328048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.847240+0000","last_change":"2024-04-13T03:31:41.849818+0000","last_active":"2024-04-13T03:31:42.847240+0000","last_peered":"2024-04-13T03:31:42.847240+0000","last_clean":"2024-04-13T03:31:42.847240+0000","last_became_active":"2024-04-13T03:31:41.848846+0000","last_became_peered":"2024-04-13T03:31:41.848846+0000","last_unstale":"2024-04-13T03:31:42.847240+0000","last_undegraded":"2024-04-13T03:31:42.847240+0000","last_fullsized":"2024-04-13T03:31:42.847240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:55:16.764935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834185+0000","last_change":"2024-04-13T03:31:37.581723+0000","last_active":"2024-04-13T03:31:42.834185+0000","last_peered":"2024-04-13T03:31:42.834185+0000","last_clean":"2024-04-13T03:31:42.834185+0000","last_became_active":"2024-04-13T03:31:37.578857+0000","last_became_peered":"2024-04-13T03:31:37.578857+0000","last_unstale":"2024-04-13T03:31:42.834185+0000","last_undegraded":"2024-04-13T03:31:42.834185+0000","last_fullsized":"2024-04-13T03:31:42.834185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:47:34.447256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.807240+0000","last_change":"2024-04-13T03:31:36.592275+0000","last_active":"2024-04-13T03:32:07.807240+0000","last_peered":"2024-04-13T03:32:07.807240+0000","last_clean":"2024-04-13T03:32:07.807240+0000","last_became_active":"2024-04-13T03:31:36.591905+0000","last_became_peered":"2024-04-13T03:31:36.591905+0000","last_unstale":"2024-04-13T03:32:07.807240+0000","last_undegraded":"2024-04-13T03:32:07.807240+0000","last_fullsized":"2024-04-13T03:32:07.807240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563104+0000","last_change":"2024-04-13T03:31:41.835412+0000","last_active":"2024-04-13T03:31:43.563104+0000","last_peered":"2024-04-13T03:31:43.563104+0000","last_clean":"2024-04-13T03:31:43.563104+0000","last_became_active":"2024-04-13T03:31:41.834978+0000","last_became_peered":"2024-04-13T03:31:41.834978+0000","last_unstale":"2024-04-13T03:31:43.563104+0000","last_undegraded":"2024-04-13T03:31:43.563104+0000","last_fullsized":"2024-04-13T03:31:43.563104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:45:52.037452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248672+0000","last_change":"2024-04-13T03:31:37.585501+0000","last_active":"2024-04-13T03:31:43.248672+0000","last_peered":"2024-04-13T03:31:43.248672+0000","last_clean":"2024-04-13T03:31:43.248672+0000","last_became_active":"2024-04-13T03:31:37.584965+0000","last_became_peered":"2024-04-13T03:31:37.584965+0000","last_unstale":"2024-04-13T03:31:43.248672+0000","last_undegraded":"2024-04-13T03:31:43.248672+0000","last_fullsized":"2024-04-13T03:31:43.248672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:44.017270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.074766+0000","last_change":"2024-04-13T03:31:36.580058+0000","last_active":"2024-04-13T03:32:06.074766+0000","last_peered":"2024-04-13T03:32:06.074766+0000","last_clean":"2024-04-13T03:32:06.074766+0000","last_became_active":"2024-04-13T03:31:36.579777+0000","last_became_peered":"2024-04-13T03:31:36.579777+0000","last_unstale":"2024-04-13T03:32:06.074766+0000","last_undegraded":"2024-04-13T03:32:06.074766+0000","last_fullsized":"2024-04-13T03:32:06.074766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:27:50.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562085+0000","last_change":"2024-04-13T03:30:17.781014+0000","last_active":"2024-04-13T03:31:43.562085+0000","last_peered":"2024-04-13T03:31:43.562085+0000","last_clean":"2024-04-13T03:31:43.562085+0000","last_became_active":"2024-04-13T03:30:17.780417+0000","last_became_peered":"2024-04-13T03:30:17.780417+0000","last_unstale":"2024-04-13T03:31:43.562085+0000","last_undegraded":"2024-04-13T03:31:43.562085+0000","last_fullsized":"2024-04-13T03:31:43.562085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:26:22.174408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:26:22.174408+0000","last_clean_scrub_stamp":"2024-04-13T03:26:22.174408+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:52:50.418399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834867+0000","last_change":"2024-04-13T03:31:41.837973+0000","last_active":"2024-04-13T03:31:42.834867+0000","last_peered":"2024-04-13T03:31:42.834867+0000","last_clean":"2024-04-13T03:31:42.834867+0000","last_became_active":"2024-04-13T03:31:41.837562+0000","last_became_peered":"2024-04-13T03:31:41.837562+0000","last_unstale":"2024-04-13T03:31:42.834867+0000","last_undegraded":"2024-04-13T03:31:42.834867+0000","last_fullsized":"2024-04-13T03:31:42.834867+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:05:09.900193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846410+0000","last_change":"2024-04-13T03:31:37.584279+0000","last_active":"2024-04-13T03:31:42.846410+0000","last_peered":"2024-04-13T03:31:42.846410+0000","last_clean":"2024-04-13T03:31:42.846410+0000","last_became_active":"2024-04-13T03:31:37.583891+0000","last_became_peered":"2024-04-13T03:31:37.583891+0000","last_unstale":"2024-04-13T03:31:42.846410+0000","last_undegraded":"2024-04-13T03:31:42.846410+0000","last_fullsized":"2024-04-13T03:31:42.846410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:56:21.446525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.043502+0000","last_change":"2024-04-13T03:31:36.561264+0000","last_active":"2024-04-13T03:32:05.043502+0000","last_peered":"2024-04-13T03:32:05.043502+0000","last_clean":"2024-04-13T03:32:05.043502+0000","last_became_active":"2024-04-13T03:31:36.560921+0000","last_became_peered":"2024-04-13T03:31:36.560921+0000","last_unstale":"2024-04-13T03:32:05.043502+0000","last_undegraded":"2024-04-13T03:32:05.043502+0000","last_fullsized":"2024-04-13T03:32:05.043502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:10.420841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563558+0000","last_change":"2024-04-13T03:31:41.833350+0000","last_active":"2024-04-13T03:31:43.563558+0000","last_peered":"2024-04-13T03:31:43.563558+0000","last_clean":"2024-04-13T03:31:43.563558+0000","last_became_active":"2024-04-13T03:31:41.832865+0000","last_became_peered":"2024-04-13T03:31:41.832865+0000","last_unstale":"2024-04-13T03:31:43.563558+0000","last_undegraded":"2024-04-13T03:31:43.563558+0000","last_fullsized":"2024-04-13T03:31:43.563558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:23:31.872562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563398+0000","last_change":"2024-04-13T03:31:36.587435+0000","last_active":"2024-04-13T03:31:43.563398+0000","last_peered":"2024-04-13T03:31:43.563398+0000","last_clean":"2024-04-13T03:31:43.563398+0000","last_became_active":"2024-04-13T03:31:36.587187+0000","last_became_peered":"2024-04-13T03:31:36.587187+0000","last_unstale":"2024-04-13T03:31:43.563398+0000","last_undegraded":"2024-04-13T03:31:43.563398+0000","last_fullsized":"2024-04-13T03:31:43.563398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:06:18.665638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846964+0000","last_change":"2024-04-13T03:31:37.584429+0000","last_active":"2024-04-13T03:31:42.846964+0000","last_peered":"2024-04-13T03:31:42.846964+0000","last_clean":"2024-04-13T03:31:42.846964+0000","last_became_active":"2024-04-13T03:31:37.583811+0000","last_became_peered":"2024-04-13T03:31:37.583811+0000","last_unstale":"2024-04-13T03:31:42.846964+0000","last_undegraded":"2024-04-13T03:31:42.846964+0000","last_fullsized":"2024-04-13T03:31:42.846964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:52:20.227001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248493+0000","last_change":"2024-04-13T03:31:41.846065+0000","last_active":"2024-04-13T03:31:43.248493+0000","last_peered":"2024-04-13T03:31:43.248493+0000","last_clean":"2024-04-13T03:31:43.248493+0000","last_became_active":"2024-04-13T03:31:41.844422+0000","last_became_peered":"2024-04-13T03:31:41.844422+0000","last_unstale":"2024-04-13T03:31:43.248493+0000","last_undegraded":"2024-04-13T03:31:43.248493+0000","last_fullsized":"2024-04-13T03:31:43.248493+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:51:41.152066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.051887+0000","last_change":"2024-04-13T03:31:36.598323+0000","last_active":"2024-04-13T03:32:05.051887+0000","last_peered":"2024-04-13T03:32:05.051887+0000","last_clean":"2024-04-13T03:32:05.051887+0000","last_became_active":"2024-04-13T03:31:36.597412+0000","last_became_peered":"2024-04-13T03:31:36.597412+0000","last_unstale":"2024-04-13T03:32:05.051887+0000","last_undegraded":"2024-04-13T03:32:05.051887+0000","last_fullsized":"2024-04-13T03:32:05.051887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:19:19.203273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.249939+0000","last_change":"2024-04-13T03:31:37.591599+0000","last_active":"2024-04-13T03:31:43.249939+0000","last_peered":"2024-04-13T03:31:43.249939+0000","last_clean":"2024-04-13T03:31:43.249939+0000","last_became_active":"2024-04-13T03:31:37.591243+0000","last_became_peered":"2024-04-13T03:31:37.591243+0000","last_unstale":"2024-04-13T03:31:43.249939+0000","last_undegraded":"2024-04-13T03:31:43.249939+0000","last_fullsized":"2024-04-13T03:31:43.249939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:11:35.460721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562028+0000","last_change":"2024-04-13T03:31:41.821604+0000","last_active":"2024-04-13T03:31:43.562028+0000","last_peered":"2024-04-13T03:31:43.562028+0000","last_clean":"2024-04-13T03:31:43.562028+0000","last_became_active":"2024-04-13T03:31:41.820804+0000","last_became_peered":"2024-04-13T03:31:41.820804+0000","last_unstale":"2024-04-13T03:31:43.562028+0000","last_undegraded":"2024-04-13T03:31:43.562028+0000","last_fullsized":"2024-04-13T03:31:43.562028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:10:45.850099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:04.039988+0000","last_change":"2024-04-13T03:31:36.582624+0000","last_active":"2024-04-13T03:32:04.039988+0000","last_peered":"2024-04-13T03:32:04.039988+0000","last_clean":"2024-04-13T03:32:04.039988+0000","last_became_active":"2024-04-13T03:31:36.582260+0000","last_became_peered":"2024-04-13T03:31:36.582260+0000","last_unstale":"2024-04-13T03:32:04.039988+0000","last_undegraded":"2024-04-13T03:32:04.039988+0000","last_fullsized":"2024-04-13T03:32:04.039988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:15.637016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562761+0000","last_change":"2024-04-13T03:31:37.591286+0000","last_active":"2024-04-13T03:31:43.562761+0000","last_peered":"2024-04-13T03:31:43.562761+0000","last_clean":"2024-04-13T03:31:43.562761+0000","last_became_active":"2024-04-13T03:31:37.590891+0000","last_became_peered":"2024-04-13T03:31:37.590891+0000","last_unstale":"2024-04-13T03:31:43.562761+0000","last_undegraded":"2024-04-13T03:31:43.562761+0000","last_fullsized":"2024-04-13T03:31:43.562761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:53:54.696082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835922+0000","last_change":"2024-04-13T03:31:41.830499+0000","last_active":"2024-04-13T03:31:42.835922+0000","last_peered":"2024-04-13T03:31:42.835922+0000","last_clean":"2024-04-13T03:31:42.835922+0000","last_became_active":"2024-04-13T03:31:41.830067+0000","last_became_peered":"2024-04-13T03:31:41.830067+0000","last_unstale":"2024-04-13T03:31:42.835922+0000","last_undegraded":"2024-04-13T03:31:42.835922+0000","last_fullsized":"2024-04-13T03:31:42.835922+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:22.635885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:04.037302+0000","last_change":"2024-04-13T03:31:36.579307+0000","last_active":"2024-04-13T03:32:04.037302+0000","last_peered":"2024-04-13T03:32:04.037302+0000","last_clean":"2024-04-13T03:32:04.037302+0000","last_became_active":"2024-04-13T03:31:36.579068+0000","last_became_peered":"2024-04-13T03:31:36.579068+0000","last_unstale":"2024-04-13T03:32:04.037302+0000","last_undegraded":"2024-04-13T03:32:04.037302+0000","last_fullsized":"2024-04-13T03:32:04.037302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:49:09.187151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562459+0000","last_change":"2024-04-13T03:31:37.575749+0000","last_active":"2024-04-13T03:31:43.562459+0000","last_peered":"2024-04-13T03:31:43.562459+0000","last_clean":"2024-04-13T03:31:43.562459+0000","last_became_active":"2024-04-13T03:31:37.574152+0000","last_became_peered":"2024-04-13T03:31:37.574152+0000","last_unstale":"2024-04-13T03:31:43.562459+0000","last_undegraded":"2024-04-13T03:31:43.562459+0000","last_fullsized":"2024-04-13T03:31:43.562459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:38:15.710978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836676+0000","last_change":"2024-04-13T03:31:41.836777+0000","last_active":"2024-04-13T03:31:42.836676+0000","last_peered":"2024-04-13T03:31:42.836676+0000","last_clean":"2024-04-13T03:31:42.836676+0000","last_became_active":"2024-04-13T03:31:41.835877+0000","last_became_peered":"2024-04-13T03:31:41.835877+0000","last_unstale":"2024-04-13T03:31:42.836676+0000","last_undegraded":"2024-04-13T03:31:42.836676+0000","last_fullsized":"2024-04-13T03:31:42.836676+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:46:44.174095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.084306+0000","last_change":"2024-04-13T03:31:36.587981+0000","last_active":"2024-04-13T03:32:07.084306+0000","last_peered":"2024-04-13T03:32:07.084306+0000","last_clean":"2024-04-13T03:32:07.084306+0000","last_became_active":"2024-04-13T03:31:36.586013+0000","last_became_peered":"2024-04-13T03:31:36.586013+0000","last_unstale":"2024-04-13T03:32:07.084306+0000","last_undegraded":"2024-04-13T03:32:07.084306+0000","last_fullsized":"2024-04-13T03:32:07.084306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:16:27.075322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562682+0000","last_change":"2024-04-13T03:31:37.594185+0000","last_active":"2024-04-13T03:31:43.562682+0000","last_peered":"2024-04-13T03:31:43.562682+0000","last_clean":"2024-04-13T03:31:43.562682+0000","last_became_active":"2024-04-13T03:31:37.593791+0000","last_became_peered":"2024-04-13T03:31:37.593791+0000","last_unstale":"2024-04-13T03:31:43.562682+0000","last_undegraded":"2024-04-13T03:31:43.562682+0000","last_fullsized":"2024-04-13T03:31:43.562682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:43:32.035371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835783+0000","last_change":"2024-04-13T03:31:41.830392+0000","last_active":"2024-04-13T03:31:42.835783+0000","last_peered":"2024-04-13T03:31:42.835783+0000","last_clean":"2024-04-13T03:31:42.835783+0000","last_became_active":"2024-04-13T03:31:41.829967+0000","last_became_peered":"2024-04-13T03:31:41.829967+0000","last_unstale":"2024-04-13T03:31:42.835783+0000","last_undegraded":"2024-04-13T03:31:42.835783+0000","last_fullsized":"2024-04-13T03:31:42.835783+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:43:25.326951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.031299+0000","last_change":"2024-04-13T03:31:36.597099+0000","last_active":"2024-04-13T03:32:05.031299+0000","last_peered":"2024-04-13T03:32:05.031299+0000","last_clean":"2024-04-13T03:32:05.031299+0000","last_became_active":"2024-04-13T03:31:36.596714+0000","last_became_peered":"2024-04-13T03:31:36.596714+0000","last_unstale":"2024-04-13T03:32:05.031299+0000","last_undegraded":"2024-04-13T03:32:05.031299+0000","last_fullsized":"2024-04-13T03:32:05.031299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:42:31.477730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.249076+0000","last_change":"2024-04-13T03:31:37.587038+0000","last_active":"2024-04-13T03:31:43.249076+0000","last_peered":"2024-04-13T03:31:43.249076+0000","last_clean":"2024-04-13T03:31:43.249076+0000","last_became_active":"2024-04-13T03:31:37.586618+0000","last_became_peered":"2024-04-13T03:31:37.586618+0000","last_unstale":"2024-04-13T03:31:43.249076+0000","last_undegraded":"2024-04-13T03:31:43.249076+0000","last_fullsized":"2024-04-13T03:31:43.249076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:57:33.868529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836017+0000","last_change":"2024-04-13T03:31:41.833148+0000","last_active":"2024-04-13T03:31:42.836017+0000","last_peered":"2024-04-13T03:31:42.836017+0000","last_clean":"2024-04-13T03:31:42.836017+0000","last_became_active":"2024-04-13T03:31:41.832714+0000","last_became_peered":"2024-04-13T03:31:41.832714+0000","last_unstale":"2024-04-13T03:31:42.836017+0000","last_undegraded":"2024-04-13T03:31:42.836017+0000","last_fullsized":"2024-04-13T03:31:42.836017+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:12:07.316488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:04.033925+0000","last_change":"2024-04-13T03:31:36.564887+0000","last_active":"2024-04-13T03:32:04.033925+0000","last_peered":"2024-04-13T03:32:04.033925+0000","last_clean":"2024-04-13T03:32:04.033925+0000","last_became_active":"2024-04-13T03:31:36.564438+0000","last_became_peered":"2024-04-13T03:31:36.564438+0000","last_unstale":"2024-04-13T03:32:04.033925+0000","last_undegraded":"2024-04-13T03:32:04.033925+0000","last_fullsized":"2024-04-13T03:32:04.033925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:19.310322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562562+0000","last_change":"2024-04-13T03:31:37.584358+0000","last_active":"2024-04-13T03:31:43.562562+0000","last_peered":"2024-04-13T03:31:43.562562+0000","last_clean":"2024-04-13T03:31:43.562562+0000","last_became_active":"2024-04-13T03:31:37.584005+0000","last_became_peered":"2024-04-13T03:31:37.584005+0000","last_unstale":"2024-04-13T03:31:43.562562+0000","last_undegraded":"2024-04-13T03:31:43.562562+0000","last_fullsized":"2024-04-13T03:31:43.562562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:07:21.465290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422621+0000","last_change":"2024-04-13T03:31:41.833451+0000","last_active":"2024-04-13T03:31:43.422621+0000","last_peered":"2024-04-13T03:31:43.422621+0000","last_clean":"2024-04-13T03:31:43.422621+0000","last_became_active":"2024-04-13T03:31:41.832734+0000","last_became_peered":"2024-04-13T03:31:41.832734+0000","last_unstale":"2024-04-13T03:31:43.422621+0000","last_undegraded":"2024-04-13T03:31:43.422621+0000","last_fullsized":"2024-04-13T03:31:43.422621+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:49:48.848152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.070843+0000","last_change":"2024-04-13T03:31:36.581697+0000","last_active":"2024-04-13T03:32:06.070843+0000","last_peered":"2024-04-13T03:32:06.070843+0000","last_clean":"2024-04-13T03:32:06.070843+0000","last_became_active":"2024-04-13T03:31:36.581345+0000","last_became_peered":"2024-04-13T03:31:36.581345+0000","last_unstale":"2024-04-13T03:32:06.070843+0000","last_undegraded":"2024-04-13T03:32:06.070843+0000","last_fullsized":"2024-04-13T03:32:06.070843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:29:53.192770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563234+0000","last_change":"2024-04-13T03:31:37.571879+0000","last_active":"2024-04-13T03:31:43.563234+0000","last_peered":"2024-04-13T03:31:43.563234+0000","last_clean":"2024-04-13T03:31:43.563234+0000","last_became_active":"2024-04-13T03:31:37.571618+0000","last_became_peered":"2024-04-13T03:31:37.571618+0000","last_unstale":"2024-04-13T03:31:43.563234+0000","last_undegraded":"2024-04-13T03:31:43.563234+0000","last_fullsized":"2024-04-13T03:31:43.563234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:42.920388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562312+0000","last_change":"2024-04-13T03:31:41.851871+0000","last_active":"2024-04-13T03:31:43.562312+0000","last_peered":"2024-04-13T03:31:43.562312+0000","last_clean":"2024-04-13T03:31:43.562312+0000","last_became_active":"2024-04-13T03:31:41.851045+0000","last_became_peered":"2024-04-13T03:31:41.851045+0000","last_unstale":"2024-04-13T03:31:43.562312+0000","last_undegraded":"2024-04-13T03:31:43.562312+0000","last_fullsized":"2024-04-13T03:31:43.562312+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:35:41.937723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.083523+0000","last_change":"2024-04-13T03:31:36.586110+0000","last_active":"2024-04-13T03:32:07.083523+0000","last_peered":"2024-04-13T03:32:07.083523+0000","last_clean":"2024-04-13T03:32:07.083523+0000","last_became_active":"2024-04-13T03:31:36.585832+0000","last_became_peered":"2024-04-13T03:31:36.585832+0000","last_unstale":"2024-04-13T03:32:07.083523+0000","last_undegraded":"2024-04-13T03:32:07.083523+0000","last_fullsized":"2024-04-13T03:32:07.083523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:00:57.825760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562919+0000","last_change":"2024-04-13T03:31:37.580004+0000","last_active":"2024-04-13T03:31:43.562919+0000","last_peered":"2024-04-13T03:31:43.562919+0000","last_clean":"2024-04-13T03:31:43.562919+0000","last_became_active":"2024-04-13T03:31:37.579634+0000","last_became_peered":"2024-04-13T03:31:37.579634+0000","last_unstale":"2024-04-13T03:31:43.562919+0000","last_undegraded":"2024-04-13T03:31:43.562919+0000","last_fullsized":"2024-04-13T03:31:43.562919+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:39:59.872588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.833865+0000","last_change":"2024-04-13T03:31:41.825456+0000","last_active":"2024-04-13T03:31:42.833865+0000","last_peered":"2024-04-13T03:31:42.833865+0000","last_clean":"2024-04-13T03:31:42.833865+0000","last_became_active":"2024-04-13T03:31:41.824929+0000","last_became_peered":"2024-04-13T03:31:41.824929+0000","last_unstale":"2024-04-13T03:31:42.833865+0000","last_undegraded":"2024-04-13T03:31:42.833865+0000","last_fullsized":"2024-04-13T03:31:42.833865+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:17:39.405147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.084252+0000","last_change":"2024-04-13T03:31:36.595009+0000","last_active":"2024-04-13T03:32:07.084252+0000","last_peered":"2024-04-13T03:32:07.084252+0000","last_clean":"2024-04-13T03:32:07.084252+0000","last_became_active":"2024-04-13T03:31:36.594314+0000","last_became_peered":"2024-04-13T03:31:36.594314+0000","last_unstale":"2024-04-13T03:32:07.084252+0000","last_undegraded":"2024-04-13T03:32:07.084252+0000","last_fullsized":"2024-04-13T03:32:07.084252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:42:33.570088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562901+0000","last_change":"2024-04-13T03:31:37.579899+0000","last_active":"2024-04-13T03:31:43.562901+0000","last_peered":"2024-04-13T03:31:43.562901+0000","last_clean":"2024-04-13T03:31:43.562901+0000","last_became_active":"2024-04-13T03:31:37.579345+0000","last_became_peered":"2024-04-13T03:31:37.579345+0000","last_unstale":"2024-04-13T03:31:43.562901+0000","last_undegraded":"2024-04-13T03:31:43.562901+0000","last_fullsized":"2024-04-13T03:31:43.562901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:06:55.646866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562802+0000","last_change":"2024-04-13T03:31:37.586373+0000","last_active":"2024-04-13T03:31:43.562802+0000","last_peered":"2024-04-13T03:31:43.562802+0000","last_clean":"2024-04-13T03:31:43.562802+0000","last_became_active":"2024-04-13T03:31:37.586061+0000","last_became_peered":"2024-04-13T03:31:37.586061+0000","last_unstale":"2024-04-13T03:31:43.562802+0000","last_undegraded":"2024-04-13T03:31:43.562802+0000","last_fullsized":"2024-04-13T03:31:43.562802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:58:52.487187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562380+0000","last_change":"2024-04-13T03:31:37.581375+0000","last_active":"2024-04-13T03:31:43.562380+0000","last_peered":"2024-04-13T03:31:43.562380+0000","last_clean":"2024-04-13T03:31:43.562380+0000","last_became_active":"2024-04-13T03:31:37.580802+0000","last_became_peered":"2024-04-13T03:31:37.580802+0000","last_unstale":"2024-04-13T03:31:43.562380+0000","last_undegraded":"2024-04-13T03:31:43.562380+0000","last_fullsized":"2024-04-13T03:31:43.562380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:41.895243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562374+0000","last_change":"2024-04-13T03:31:37.580598+0000","last_active":"2024-04-13T03:31:43.562374+0000","last_peered":"2024-04-13T03:31:43.562374+0000","last_clean":"2024-04-13T03:31:43.562374+0000","last_became_active":"2024-04-13T03:31:37.580276+0000","last_became_peered":"2024-04-13T03:31:37.580276+0000","last_unstale":"2024-04-13T03:31:43.562374+0000","last_undegraded":"2024-04-13T03:31:43.562374+0000","last_fullsized":"2024-04-13T03:31:43.562374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:56:02.302300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562697+0000","last_change":"2024-04-13T03:31:37.596435+0000","last_active":"2024-04-13T03:31:43.562697+0000","last_peered":"2024-04-13T03:31:43.562697+0000","last_clean":"2024-04-13T03:31:43.562697+0000","last_became_active":"2024-04-13T03:31:37.596095+0000","last_became_peered":"2024-04-13T03:31:37.596095+0000","last_unstale":"2024-04-13T03:31:43.562697+0000","last_undegraded":"2024-04-13T03:31:43.562697+0000","last_fullsized":"2024-04-13T03:31:43.562697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:04.425037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846887+0000","last_change":"2024-04-13T03:31:37.578003+0000","last_active":"2024-04-13T03:31:42.846887+0000","last_peered":"2024-04-13T03:31:42.846887+0000","last_clean":"2024-04-13T03:31:42.846887+0000","last_became_active":"2024-04-13T03:31:37.577637+0000","last_became_peered":"2024-04-13T03:31:37.577637+0000","last_unstale":"2024-04-13T03:31:42.846887+0000","last_undegraded":"2024-04-13T03:31:42.846887+0000","last_fullsized":"2024-04-13T03:31:42.846887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:30:37.138611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563816+0000","last_change":"2024-04-13T03:31:37.585404+0000","last_active":"2024-04-13T03:31:43.563816+0000","last_peered":"2024-04-13T03:31:43.563816+0000","last_clean":"2024-04-13T03:31:43.563816+0000","last_became_active":"2024-04-13T03:31:37.585060+0000","last_became_peered":"2024-04-13T03:31:37.585060+0000","last_unstale":"2024-04-13T03:31:43.563816+0000","last_undegraded":"2024-04-13T03:31:43.563816+0000","last_fullsized":"2024-04-13T03:31:43.563816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:37:52.165706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563001+0000","last_change":"2024-04-13T03:31:37.576265+0000","last_active":"2024-04-13T03:31:43.563001+0000","last_peered":"2024-04-13T03:31:43.563001+0000","last_clean":"2024-04-13T03:31:43.563001+0000","last_became_active":"2024-04-13T03:31:37.575906+0000","last_became_peered":"2024-04-13T03:31:37.575906+0000","last_unstale":"2024-04-13T03:31:43.563001+0000","last_undegraded":"2024-04-13T03:31:43.563001+0000","last_fullsized":"2024-04-13T03:31:43.563001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:31:34.505830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.250200+0000","last_change":"2024-04-13T03:31:37.582865+0000","last_active":"2024-04-13T03:31:43.250200+0000","last_peered":"2024-04-13T03:31:43.250200+0000","last_clean":"2024-04-13T03:31:43.250200+0000","last_became_active":"2024-04-13T03:31:37.581055+0000","last_became_peered":"2024-04-13T03:31:37.581055+0000","last_unstale":"2024-04-13T03:31:43.250200+0000","last_undegraded":"2024-04-13T03:31:43.250200+0000","last_fullsized":"2024-04-13T03:31:43.250200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:01:07.815483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834009+0000","last_change":"2024-04-13T03:31:37.587677+0000","last_active":"2024-04-13T03:31:42.834009+0000","last_peered":"2024-04-13T03:31:42.834009+0000","last_clean":"2024-04-13T03:31:42.834009+0000","last_became_active":"2024-04-13T03:31:37.586778+0000","last_became_peered":"2024-04-13T03:31:37.586778+0000","last_unstale":"2024-04-13T03:31:42.834009+0000","last_undegraded":"2024-04-13T03:31:42.834009+0000","last_fullsized":"2024-04-13T03:31:42.834009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:52:13.968559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248625+0000","last_change":"2024-04-13T03:31:37.577694+0000","last_active":"2024-04-13T03:31:43.248625+0000","last_peered":"2024-04-13T03:31:43.248625+0000","last_clean":"2024-04-13T03:31:43.248625+0000","last_became_active":"2024-04-13T03:31:37.577287+0000","last_became_peered":"2024-04-13T03:31:37.577287+0000","last_unstale":"2024-04-13T03:31:43.248625+0000","last_undegraded":"2024-04-13T03:31:43.248625+0000","last_fullsized":"2024-04-13T03:31:43.248625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:16:26.503330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422645+0000","last_change":"2024-04-13T03:31:37.591160+0000","last_active":"2024-04-13T03:31:43.422645+0000","last_peered":"2024-04-13T03:31:43.422645+0000","last_clean":"2024-04-13T03:31:43.422645+0000","last_became_active":"2024-04-13T03:31:37.590666+0000","last_became_peered":"2024-04-13T03:31:37.590666+0000","last_unstale":"2024-04-13T03:31:43.422645+0000","last_undegraded":"2024-04-13T03:31:43.422645+0000","last_fullsized":"2024-04-13T03:31:43.422645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:43:25.976108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846756+0000","last_change":"2024-04-13T03:31:37.594433+0000","last_active":"2024-04-13T03:31:42.846756+0000","last_peered":"2024-04-13T03:31:42.846756+0000","last_clean":"2024-04-13T03:31:42.846756+0000","last_became_active":"2024-04-13T03:31:37.590942+0000","last_became_peered":"2024-04-13T03:31:37.590942+0000","last_unstale":"2024-04-13T03:31:42.846756+0000","last_undegraded":"2024-04-13T03:31:42.846756+0000","last_fullsized":"2024-04-13T03:31:42.846756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:42:47.534702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563560+0000","last_change":"2024-04-13T03:31:37.583342+0000","last_active":"2024-04-13T03:31:43.563560+0000","last_peered":"2024-04-13T03:31:43.563560+0000","last_clean":"2024-04-13T03:31:43.563560+0000","last_became_active":"2024-04-13T03:31:37.582983+0000","last_became_peered":"2024-04-13T03:31:37.582983+0000","last_unstale":"2024-04-13T03:31:43.563560+0000","last_undegraded":"2024-04-13T03:31:43.563560+0000","last_fullsized":"2024-04-13T03:31:43.563560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:24:26.616843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284330,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":47335,"num_objects":77,"num_object_clones":0,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":22,"num_read_kb":57,"num_write":177,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":113697,"data_compressed":3654,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350996,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":489158,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612633,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1118208,"data_stored":1118271,"data_compressed":52693,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":484688,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":484608,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1077484,"data_compressed":50866,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":484592,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":508763,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":486048,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":489684,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":484300,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920826,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":484536,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1081545,"data_compressed":50866,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":43658,"data_compressed":1827,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":44033,"data_compressed":1827,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T03:32:10.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-13T03:32:10.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:10 smithi119 ceph-mon[35835]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:32:10.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:10 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1396673382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:32:10.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:10 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1544435700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.691+0000 7f51e2400640 1 -- 172.21.15.16:0/1717104395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc078560 msgr2=0x7f51dc078e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.691+0000 7f51e2400640 1 --2- 172.21.15.16:0/1717104395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc078560 0x7f51dc078e00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009900 tx=0x7f51d402bc40 comp rx=0 tx=0).stop 2024-04-13T03:32:10.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.691+0000 7f51e2400640 1 -- 172.21.15.16:0/1717104395 shutdown_connections 2024-04-13T03:32:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.691+0000 7f51e2400640 1 --2- 172.21.15.16:0/1717104395 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc078560 0x7f51dc078e00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.691+0000 7f51e2400640 1 --2- 172.21.15.16:0/1717104395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51dc077bc0 0x7f51dc078020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.691+0000 7f51e2400640 1 --2- 172.21.15.16:0/1717104395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51dc076980 0x7f51dc076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.691+0000 7f51e2400640 1 -- 172.21.15.16:0/1717104395 >> 172.21.15.16:0/1717104395 conn(0x7f51dc0ff7b0 msgr2=0x7f51dc101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.692+0000 7f51e2400640 1 -- 172.21.15.16:0/1717104395 shutdown_connections 2024-04-13T03:32:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.692+0000 7f51e2400640 1 -- 172.21.15.16:0/1717104395 wait complete. 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.693+0000 7f51e2400640 1 Processor -- start 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.693+0000 7f51e2400640 1 -- start start 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.693+0000 7f51e2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc076980 0x7f51dc19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.693+0000 7f51e2400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51dc077bc0 0x7f51dc19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.693+0000 7f51e2400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51dc078560 0x7f51dc1a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.693+0000 7f51e2400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc1142a0 con 0x7f51dc076980 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.693+0000 7f51e2400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc114120 con 0x7f51dc078560 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.693+0000 7f51e2400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc114420 con 0x7f51dc077bc0 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51dbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51dc078560 0x7f51dc1a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51dbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51dc078560 0x7f51dc1a5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:47194/0 (socket says 172.21.15.16:47194) 2024-04-13T03:32:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51dbe00640 1 -- 172.21.15.16:0/2823563176 learned_addr learned my addr 172.21.15.16:0/2823563176 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51daa00640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51dc077bc0 0x7f51dc19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51db400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc076980 0x7f51dc19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51dbe00640 1 -- 172.21.15.16:0/2823563176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51dc077bc0 msgr2=0x7f51dc19ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51dbe00640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51dc077bc0 0x7f51dc19ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51dbe00640 1 -- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc076980 msgr2=0x7f51dc19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51dbe00640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc076980 0x7f51dc19e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51dbe00640 1 -- 172.21.15.16:0/2823563176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51dc1a64c0 con 0x7f51dc078560 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51db400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc076980 0x7f51dc19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.694+0000 7f51dbe00640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51dc078560 0x7f51dc1a5e50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f51d4005ab0 tx=0x7f51d402c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.695+0000 7f51d3e00640 1 -- 172.21.15.16:0/2823563176 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4041070 con 0x7f51dc078560 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.695+0000 7f51d3e00640 1 -- 172.21.15.16:0/2823563176 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d4004450 con 0x7f51dc078560 2024-04-13T03:32:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.695+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51dc1a6750 con 0x7f51dc078560 2024-04-13T03:32:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.695+0000 7f51d3e00640 1 -- 172.21.15.16:0/2823563176 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4003e40 con 0x7f51dc078560 2024-04-13T03:32:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.695+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51dc1a6c40 con 0x7f51dc078560 2024-04-13T03:32:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.696+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51dc110cf0 con 0x7f51dc078560 2024-04-13T03:32:10.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.698+0000 7f51d3e00640 1 -- 172.21.15.16:0/2823563176 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f51d4044070 con 0x7f51dc078560 2024-04-13T03:32:10.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.699+0000 7f51d3e00640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f51c00a3c40 0x7f51c00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.699+0000 7f51d3e00640 1 -- 172.21.15.16:0/2823563176 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f51d40f2600 con 0x7f51dc078560 2024-04-13T03:32:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.700+0000 7f51db400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f51c00a3c40 0x7f51c00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.701+0000 7f51db400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f51c00a3c40 0x7f51c00a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f51cc005fd0 tx=0x7f51cc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.702+0000 7f51d3e00640 1 -- 172.21.15.16:0/2823563176 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51d40becf0 con 0x7f51dc078560 2024-04-13T03:32:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.909+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f51dc062400 con 0x7f51c00a3c40 2024-04-13T03:32:10.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.940+0000 7f51d3e00640 1 -- 172.21.15.16:0/2823563176 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340692 (secure 0 0 0) 0x7f51dc062400 con 0x7f51c00a3c40 2024-04-13T03:32:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:32:10.948 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-13T03:32:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f51c00a3c40 msgr2=0x7f51c00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f51c00a3c40 0x7f51c00a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f51cc005fd0 tx=0x7f51cc009500 comp rx=0 tx=0).stop 2024-04-13T03:32:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51dc078560 msgr2=0x7f51dc1a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51dc078560 0x7f51dc1a5e50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f51d4005ab0 tx=0x7f51d402c6d0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 shutdown_connections 2024-04-13T03:32:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f51c00a3c40 0x7f51c00a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51dc078560 0x7f51dc1a5e50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f51dc077bc0 0x7f51dc19ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 --2- 172.21.15.16:0/2823563176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51dc076980 0x7f51dc19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.948+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 >> 172.21.15.16:0/2823563176 conn(0x7f51dc0ff7b0 msgr2=0x7f51dc100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.949+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 shutdown_connections 2024-04-13T03:32:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:10.949+0000 7f51e2400640 1 -- 172.21.15.16:0/2823563176 wait complete. 2024-04-13T03:32:10.972 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-04-13T03:32:09.879263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637703,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":158,"num_read_kb":176,"num_write":358,"num_write_kb":3044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":294,"ondisk_log_size":294,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369448,"kb_used_data":11976,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572376,"statfs":{"total":1151940427776,"available":1151562113024,"internally_reserved":0,"allocated":12263424,"data_stored":7673677,"data_compressed":286806,"data_compressed_allocated":385024,"data_compressed_original":2269184,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44476,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":22,"num_read_kb":57,"num_write":130,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005919"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563991+0000","last_change":"2024-04-13T03:31:37.597340+0000","last_active":"2024-04-13T03:31:43.563991+0000","last_peered":"2024-04-13T03:31:43.563991+0000","last_clean":"2024-04-13T03:31:43.563991+0000","last_became_active":"2024-04-13T03:31:37.597081+0000","last_became_peered":"2024-04-13T03:31:37.597081+0000","last_unstale":"2024-04-13T03:31:43.563991+0000","last_undegraded":"2024-04-13T03:31:43.563991+0000","last_fullsized":"2024-04-13T03:31:43.563991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:10:36.679921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562649+0000","last_change":"2024-04-13T03:31:37.578150+0000","last_active":"2024-04-13T03:31:43.562649+0000","last_peered":"2024-04-13T03:31:43.562649+0000","last_clean":"2024-04-13T03:31:43.562649+0000","last_became_active":"2024-04-13T03:31:37.574207+0000","last_became_peered":"2024-04-13T03:31:37.574207+0000","last_unstale":"2024-04-13T03:31:43.562649+0000","last_undegraded":"2024-04-13T03:31:43.562649+0000","last_fullsized":"2024-04-13T03:31:43.562649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:38:41.999046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562348+0000","last_change":"2024-04-13T03:31:37.581782+0000","last_active":"2024-04-13T03:31:43.562348+0000","last_peered":"2024-04-13T03:31:43.562348+0000","last_clean":"2024-04-13T03:31:43.562348+0000","last_became_active":"2024-04-13T03:31:37.581448+0000","last_became_peered":"2024-04-13T03:31:37.581448+0000","last_unstale":"2024-04-13T03:31:43.562348+0000","last_undegraded":"2024-04-13T03:31:43.562348+0000","last_fullsized":"2024-04-13T03:31:43.562348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:56:44.538333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836121+0000","last_change":"2024-04-13T03:31:37.578982+0000","last_active":"2024-04-13T03:31:42.836121+0000","last_peered":"2024-04-13T03:31:42.836121+0000","last_clean":"2024-04-13T03:31:42.836121+0000","last_became_active":"2024-04-13T03:31:37.578375+0000","last_became_peered":"2024-04-13T03:31:37.578375+0000","last_unstale":"2024-04-13T03:31:42.836121+0000","last_undegraded":"2024-04-13T03:31:42.836121+0000","last_fullsized":"2024-04-13T03:31:42.836121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:13:36.312872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562738+0000","last_change":"2024-04-13T03:31:37.577546+0000","last_active":"2024-04-13T03:31:43.562738+0000","last_peered":"2024-04-13T03:31:43.562738+0000","last_clean":"2024-04-13T03:31:43.562738+0000","last_became_active":"2024-04-13T03:31:37.577238+0000","last_became_peered":"2024-04-13T03:31:37.577238+0000","last_unstale":"2024-04-13T03:31:43.562738+0000","last_undegraded":"2024-04-13T03:31:43.562738+0000","last_fullsized":"2024-04-13T03:31:43.562738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:25:17.759986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562623+0000","last_change":"2024-04-13T03:31:37.585806+0000","last_active":"2024-04-13T03:31:43.562623+0000","last_peered":"2024-04-13T03:31:43.562623+0000","last_clean":"2024-04-13T03:31:43.562623+0000","last_became_active":"2024-04-13T03:31:37.585448+0000","last_became_peered":"2024-04-13T03:31:37.585448+0000","last_unstale":"2024-04-13T03:31:43.562623+0000","last_undegraded":"2024-04-13T03:31:43.562623+0000","last_fullsized":"2024-04-13T03:31:43.562623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:32:19.336091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562233+0000","last_change":"2024-04-13T03:31:37.579546+0000","last_active":"2024-04-13T03:31:43.562233+0000","last_peered":"2024-04-13T03:31:43.562233+0000","last_clean":"2024-04-13T03:31:43.562233+0000","last_became_active":"2024-04-13T03:31:37.579165+0000","last_became_peered":"2024-04-13T03:31:37.579165+0000","last_unstale":"2024-04-13T03:31:43.562233+0000","last_undegraded":"2024-04-13T03:31:43.562233+0000","last_fullsized":"2024-04-13T03:31:43.562233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:55:51.788344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248239+0000","last_change":"2024-04-13T03:31:37.587374+0000","last_active":"2024-04-13T03:31:43.248239+0000","last_peered":"2024-04-13T03:31:43.248239+0000","last_clean":"2024-04-13T03:31:43.248239+0000","last_became_active":"2024-04-13T03:31:37.586998+0000","last_became_peered":"2024-04-13T03:31:37.586998+0000","last_unstale":"2024-04-13T03:31:43.248239+0000","last_undegraded":"2024-04-13T03:31:43.248239+0000","last_fullsized":"2024-04-13T03:31:43.248239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:16:46.441001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422978+0000","last_change":"2024-04-13T03:31:37.580579+0000","last_active":"2024-04-13T03:31:43.422978+0000","last_peered":"2024-04-13T03:31:43.422978+0000","last_clean":"2024-04-13T03:31:43.422978+0000","last_became_active":"2024-04-13T03:31:37.580243+0000","last_became_peered":"2024-04-13T03:31:37.580243+0000","last_unstale":"2024-04-13T03:31:43.422978+0000","last_undegraded":"2024-04-13T03:31:43.422978+0000","last_fullsized":"2024-04-13T03:31:43.422978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:42.080260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562737+0000","last_change":"2024-04-13T03:31:37.567731+0000","last_active":"2024-04-13T03:31:43.562737+0000","last_peered":"2024-04-13T03:31:43.562737+0000","last_clean":"2024-04-13T03:31:43.562737+0000","last_became_active":"2024-04-13T03:31:37.567402+0000","last_became_peered":"2024-04-13T03:31:37.567402+0000","last_unstale":"2024-04-13T03:31:43.562737+0000","last_undegraded":"2024-04-13T03:31:43.562737+0000","last_fullsized":"2024-04-13T03:31:43.562737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:54:46.976698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563160+0000","last_change":"2024-04-13T03:31:37.576379+0000","last_active":"2024-04-13T03:31:43.563160+0000","last_peered":"2024-04-13T03:31:43.563160+0000","last_clean":"2024-04-13T03:31:43.563160+0000","last_became_active":"2024-04-13T03:31:37.576062+0000","last_became_peered":"2024-04-13T03:31:37.576062+0000","last_unstale":"2024-04-13T03:31:43.563160+0000","last_undegraded":"2024-04-13T03:31:43.563160+0000","last_fullsized":"2024-04-13T03:31:43.563160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:21:17.507656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835986+0000","last_change":"2024-04-13T03:31:37.578667+0000","last_active":"2024-04-13T03:31:42.835986+0000","last_peered":"2024-04-13T03:31:42.835986+0000","last_clean":"2024-04-13T03:31:42.835986+0000","last_became_active":"2024-04-13T03:31:37.577794+0000","last_became_peered":"2024-04-13T03:31:37.577794+0000","last_unstale":"2024-04-13T03:31:42.835986+0000","last_undegraded":"2024-04-13T03:31:42.835986+0000","last_fullsized":"2024-04-13T03:31:42.835986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:48:51.858309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846709+0000","last_change":"2024-04-13T03:31:37.576827+0000","last_active":"2024-04-13T03:31:42.846709+0000","last_peered":"2024-04-13T03:31:42.846709+0000","last_clean":"2024-04-13T03:31:42.846709+0000","last_became_active":"2024-04-13T03:31:37.576418+0000","last_became_peered":"2024-04-13T03:31:37.576418+0000","last_unstale":"2024-04-13T03:31:42.846709+0000","last_undegraded":"2024-04-13T03:31:42.846709+0000","last_fullsized":"2024-04-13T03:31:42.846709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:45:22.462714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562207+0000","last_change":"2024-04-13T03:31:37.577623+0000","last_active":"2024-04-13T03:31:43.562207+0000","last_peered":"2024-04-13T03:31:43.562207+0000","last_clean":"2024-04-13T03:31:43.562207+0000","last_became_active":"2024-04-13T03:31:37.577175+0000","last_became_peered":"2024-04-13T03:31:37.577175+0000","last_unstale":"2024-04-13T03:31:43.562207+0000","last_undegraded":"2024-04-13T03:31:43.562207+0000","last_fullsized":"2024-04-13T03:31:43.562207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:52:29.604492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562093+0000","last_change":"2024-04-13T03:31:37.581036+0000","last_active":"2024-04-13T03:31:43.562093+0000","last_peered":"2024-04-13T03:31:43.562093+0000","last_clean":"2024-04-13T03:31:43.562093+0000","last_became_active":"2024-04-13T03:31:37.580600+0000","last_became_peered":"2024-04-13T03:31:37.580600+0000","last_unstale":"2024-04-13T03:31:43.562093+0000","last_undegraded":"2024-04-13T03:31:43.562093+0000","last_fullsized":"2024-04-13T03:31:43.562093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:12:29.001001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422361+0000","last_change":"2024-04-13T03:31:37.581073+0000","last_active":"2024-04-13T03:31:43.422361+0000","last_peered":"2024-04-13T03:31:43.422361+0000","last_clean":"2024-04-13T03:31:43.422361+0000","last_became_active":"2024-04-13T03:31:37.580719+0000","last_became_peered":"2024-04-13T03:31:37.580719+0000","last_unstale":"2024-04-13T03:31:43.422361+0000","last_undegraded":"2024-04-13T03:31:43.422361+0000","last_fullsized":"2024-04-13T03:31:43.422361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:22.542263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563743+0000","last_change":"2024-04-13T03:31:37.597572+0000","last_active":"2024-04-13T03:31:43.563743+0000","last_peered":"2024-04-13T03:31:43.563743+0000","last_clean":"2024-04-13T03:31:43.563743+0000","last_became_active":"2024-04-13T03:31:37.597244+0000","last_became_peered":"2024-04-13T03:31:37.597244+0000","last_unstale":"2024-04-13T03:31:43.563743+0000","last_undegraded":"2024-04-13T03:31:43.563743+0000","last_fullsized":"2024-04-13T03:31:43.563743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:39:13.365427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835972+0000","last_change":"2024-04-13T03:31:37.596749+0000","last_active":"2024-04-13T03:31:42.835972+0000","last_peered":"2024-04-13T03:31:42.835972+0000","last_clean":"2024-04-13T03:31:42.835972+0000","last_became_active":"2024-04-13T03:31:37.592409+0000","last_became_peered":"2024-04-13T03:31:37.592409+0000","last_unstale":"2024-04-13T03:31:42.835972+0000","last_undegraded":"2024-04-13T03:31:42.835972+0000","last_fullsized":"2024-04-13T03:31:42.835972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:35:22.360224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563035+0000","last_change":"2024-04-13T03:31:37.571354+0000","last_active":"2024-04-13T03:31:43.563035+0000","last_peered":"2024-04-13T03:31:43.563035+0000","last_clean":"2024-04-13T03:31:43.563035+0000","last_became_active":"2024-04-13T03:31:37.571012+0000","last_became_peered":"2024-04-13T03:31:37.571012+0000","last_unstale":"2024-04-13T03:31:43.563035+0000","last_undegraded":"2024-04-13T03:31:43.563035+0000","last_fullsized":"2024-04-13T03:31:43.563035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:50:26.492090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562015+0000","last_change":"2024-04-13T03:31:41.844448+0000","last_active":"2024-04-13T03:31:43.562015+0000","last_peered":"2024-04-13T03:31:43.562015+0000","last_clean":"2024-04-13T03:31:43.562015+0000","last_became_active":"2024-04-13T03:31:41.844143+0000","last_became_peered":"2024-04-13T03:31:41.844143+0000","last_unstale":"2024-04-13T03:31:43.562015+0000","last_undegraded":"2024-04-13T03:31:43.562015+0000","last_fullsized":"2024-04-13T03:31:43.562015+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:26:40.811950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.055705+0000","last_change":"2024-04-13T03:31:36.581372+0000","last_active":"2024-04-13T03:32:05.055705+0000","last_peered":"2024-04-13T03:32:05.055705+0000","last_clean":"2024-04-13T03:32:05.055705+0000","last_became_active":"2024-04-13T03:31:36.580599+0000","last_became_peered":"2024-04-13T03:31:36.580599+0000","last_unstale":"2024-04-13T03:32:05.055705+0000","last_undegraded":"2024-04-13T03:32:05.055705+0000","last_fullsized":"2024-04-13T03:32:05.055705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:10:09.839049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.249836+0000","last_change":"2024-04-13T03:31:37.584511+0000","last_active":"2024-04-13T03:31:43.249836+0000","last_peered":"2024-04-13T03:31:43.249836+0000","last_clean":"2024-04-13T03:31:43.249836+0000","last_became_active":"2024-04-13T03:31:37.584094+0000","last_became_peered":"2024-04-13T03:31:37.584094+0000","last_unstale":"2024-04-13T03:31:43.249836+0000","last_undegraded":"2024-04-13T03:31:43.249836+0000","last_fullsized":"2024-04-13T03:31:43.249836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:47:36.163763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835423+0000","last_change":"2024-04-13T03:31:41.834169+0000","last_active":"2024-04-13T03:31:42.835423+0000","last_peered":"2024-04-13T03:31:42.835423+0000","last_clean":"2024-04-13T03:31:42.835423+0000","last_became_active":"2024-04-13T03:31:41.833545+0000","last_became_peered":"2024-04-13T03:31:41.833545+0000","last_unstale":"2024-04-13T03:31:42.835423+0000","last_undegraded":"2024-04-13T03:31:42.835423+0000","last_fullsized":"2024-04-13T03:31:42.835423+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:37:29.657978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.249994+0000","last_change":"2024-04-13T03:31:36.596225+0000","last_active":"2024-04-13T03:31:43.249994+0000","last_peered":"2024-04-13T03:31:43.249994+0000","last_clean":"2024-04-13T03:31:43.249994+0000","last_became_active":"2024-04-13T03:31:36.595520+0000","last_became_peered":"2024-04-13T03:31:36.595520+0000","last_unstale":"2024-04-13T03:31:43.249994+0000","last_undegraded":"2024-04-13T03:31:43.249994+0000","last_fullsized":"2024-04-13T03:31:43.249994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:15:35.172065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562506+0000","last_change":"2024-04-13T03:31:37.587264+0000","last_active":"2024-04-13T03:31:43.562506+0000","last_peered":"2024-04-13T03:31:43.562506+0000","last_clean":"2024-04-13T03:31:43.562506+0000","last_became_active":"2024-04-13T03:31:37.585574+0000","last_became_peered":"2024-04-13T03:31:37.585574+0000","last_unstale":"2024-04-13T03:31:43.562506+0000","last_undegraded":"2024-04-13T03:31:43.562506+0000","last_fullsized":"2024-04-13T03:31:43.562506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:39.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835558+0000","last_change":"2024-04-13T03:31:41.827237+0000","last_active":"2024-04-13T03:31:42.835558+0000","last_peered":"2024-04-13T03:31:42.835558+0000","last_clean":"2024-04-13T03:31:42.835558+0000","last_became_active":"2024-04-13T03:31:41.826799+0000","last_became_peered":"2024-04-13T03:31:41.826799+0000","last_unstale":"2024-04-13T03:31:42.835558+0000","last_undegraded":"2024-04-13T03:31:42.835558+0000","last_fullsized":"2024-04-13T03:31:42.835558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:26:14.337654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.043515+0000","last_change":"2024-04-13T03:31:36.564752+0000","last_active":"2024-04-13T03:32:05.043515+0000","last_peered":"2024-04-13T03:32:05.043515+0000","last_clean":"2024-04-13T03:32:05.043515+0000","last_became_active":"2024-04-13T03:31:36.564413+0000","last_became_peered":"2024-04-13T03:31:36.564413+0000","last_unstale":"2024-04-13T03:32:05.043515+0000","last_undegraded":"2024-04-13T03:32:05.043515+0000","last_fullsized":"2024-04-13T03:32:05.043515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:34:52.369875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422639+0000","last_change":"2024-04-13T03:31:37.579316+0000","last_active":"2024-04-13T03:31:43.422639+0000","last_peered":"2024-04-13T03:31:43.422639+0000","last_clean":"2024-04-13T03:31:43.422639+0000","last_became_active":"2024-04-13T03:31:37.578971+0000","last_became_peered":"2024-04-13T03:31:37.578971+0000","last_unstale":"2024-04-13T03:31:43.422639+0000","last_undegraded":"2024-04-13T03:31:43.422639+0000","last_fullsized":"2024-04-13T03:31:43.422639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:54:57.013722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835551+0000","last_change":"2024-04-13T03:31:41.841120+0000","last_active":"2024-04-13T03:31:42.835551+0000","last_peered":"2024-04-13T03:31:42.835551+0000","last_clean":"2024-04-13T03:31:42.835551+0000","last_became_active":"2024-04-13T03:31:41.840833+0000","last_became_peered":"2024-04-13T03:31:41.840833+0000","last_unstale":"2024-04-13T03:31:42.835551+0000","last_undegraded":"2024-04-13T03:31:42.835551+0000","last_fullsized":"2024-04-13T03:31:42.835551+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:13:09.979055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563354+0000","last_change":"2024-04-13T03:31:36.561428+0000","last_active":"2024-04-13T03:31:43.563354+0000","last_peered":"2024-04-13T03:31:43.563354+0000","last_clean":"2024-04-13T03:31:43.563354+0000","last_became_active":"2024-04-13T03:31:36.561100+0000","last_became_peered":"2024-04-13T03:31:36.561100+0000","last_unstale":"2024-04-13T03:31:43.563354+0000","last_undegraded":"2024-04-13T03:31:43.563354+0000","last_fullsized":"2024-04-13T03:31:43.563354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:04:32.125292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834499+0000","last_change":"2024-04-13T03:31:37.584932+0000","last_active":"2024-04-13T03:31:42.834499+0000","last_peered":"2024-04-13T03:31:42.834499+0000","last_clean":"2024-04-13T03:31:42.834499+0000","last_became_active":"2024-04-13T03:31:37.584607+0000","last_became_peered":"2024-04-13T03:31:37.584607+0000","last_unstale":"2024-04-13T03:31:42.834499+0000","last_undegraded":"2024-04-13T03:31:42.834499+0000","last_fullsized":"2024-04-13T03:31:42.834499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:15:50.573605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562972+0000","last_change":"2024-04-13T03:31:41.824295+0000","last_active":"2024-04-13T03:31:43.562972+0000","last_peered":"2024-04-13T03:31:43.562972+0000","last_clean":"2024-04-13T03:31:43.562972+0000","last_became_active":"2024-04-13T03:31:41.823810+0000","last_became_peered":"2024-04-13T03:31:41.823810+0000","last_unstale":"2024-04-13T03:31:43.562972+0000","last_undegraded":"2024-04-13T03:31:43.562972+0000","last_fullsized":"2024-04-13T03:31:43.562972+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:40:08.888519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.814594+0000","last_change":"2024-04-13T03:31:36.558866+0000","last_active":"2024-04-13T03:32:07.814594+0000","last_peered":"2024-04-13T03:32:07.814594+0000","last_clean":"2024-04-13T03:32:07.814594+0000","last_became_active":"2024-04-13T03:31:36.558541+0000","last_became_peered":"2024-04-13T03:31:36.558541+0000","last_unstale":"2024-04-13T03:32:07.814594+0000","last_undegraded":"2024-04-13T03:32:07.814594+0000","last_fullsized":"2024-04-13T03:32:07.814594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:59:29.431084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563115+0000","last_change":"2024-04-13T03:31:37.581298+0000","last_active":"2024-04-13T03:31:43.563115+0000","last_peered":"2024-04-13T03:31:43.563115+0000","last_clean":"2024-04-13T03:31:43.563115+0000","last_became_active":"2024-04-13T03:31:37.580963+0000","last_became_peered":"2024-04-13T03:31:37.580963+0000","last_unstale":"2024-04-13T03:31:43.563115+0000","last_undegraded":"2024-04-13T03:31:43.563115+0000","last_fullsized":"2024-04-13T03:31:43.563115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:47:27.921928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563122+0000","last_change":"2024-04-13T03:31:41.831221+0000","last_active":"2024-04-13T03:31:43.563122+0000","last_peered":"2024-04-13T03:31:43.563122+0000","last_clean":"2024-04-13T03:31:43.563122+0000","last_became_active":"2024-04-13T03:31:41.830246+0000","last_became_peered":"2024-04-13T03:31:41.830246+0000","last_unstale":"2024-04-13T03:31:43.563122+0000","last_undegraded":"2024-04-13T03:31:43.563122+0000","last_fullsized":"2024-04-13T03:31:43.563122+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:08:16.624658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.074395+0000","last_change":"2024-04-13T03:31:36.585534+0000","last_active":"2024-04-13T03:32:06.074395+0000","last_peered":"2024-04-13T03:32:06.074395+0000","last_clean":"2024-04-13T03:32:06.074395+0000","last_became_active":"2024-04-13T03:31:36.585324+0000","last_became_peered":"2024-04-13T03:31:36.585324+0000","last_unstale":"2024-04-13T03:32:06.074395+0000","last_undegraded":"2024-04-13T03:32:06.074395+0000","last_fullsized":"2024-04-13T03:32:06.074395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:12:35.971994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834461+0000","last_change":"2024-04-13T03:31:37.587440+0000","last_active":"2024-04-13T03:31:42.834461+0000","last_peered":"2024-04-13T03:31:42.834461+0000","last_clean":"2024-04-13T03:31:42.834461+0000","last_became_active":"2024-04-13T03:31:37.581413+0000","last_became_peered":"2024-04-13T03:31:37.581413+0000","last_unstale":"2024-04-13T03:31:42.834461+0000","last_undegraded":"2024-04-13T03:31:42.834461+0000","last_fullsized":"2024-04-13T03:31:42.834461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:14:26.066107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563930+0000","last_change":"2024-04-13T03:31:41.848451+0000","last_active":"2024-04-13T03:31:43.563930+0000","last_peered":"2024-04-13T03:31:43.563930+0000","last_clean":"2024-04-13T03:31:43.563930+0000","last_became_active":"2024-04-13T03:31:41.848019+0000","last_became_peered":"2024-04-13T03:31:41.848019+0000","last_unstale":"2024-04-13T03:31:43.563930+0000","last_undegraded":"2024-04-13T03:31:43.563930+0000","last_fullsized":"2024-04-13T03:31:43.563930+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:26:34.558142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.055541+0000","last_change":"2024-04-13T03:31:36.588393+0000","last_active":"2024-04-13T03:32:05.055541+0000","last_peered":"2024-04-13T03:32:05.055541+0000","last_clean":"2024-04-13T03:32:05.055541+0000","last_became_active":"2024-04-13T03:31:36.587184+0000","last_became_peered":"2024-04-13T03:31:36.587184+0000","last_unstale":"2024-04-13T03:32:05.055541+0000","last_undegraded":"2024-04-13T03:32:05.055541+0000","last_fullsized":"2024-04-13T03:32:05.055541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:44:24.985634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248478+0000","last_change":"2024-04-13T03:31:37.600417+0000","last_active":"2024-04-13T03:31:43.248478+0000","last_peered":"2024-04-13T03:31:43.248478+0000","last_clean":"2024-04-13T03:31:43.248478+0000","last_became_active":"2024-04-13T03:31:37.599889+0000","last_became_peered":"2024-04-13T03:31:37.599889+0000","last_unstale":"2024-04-13T03:31:43.248478+0000","last_undegraded":"2024-04-13T03:31:43.248478+0000","last_fullsized":"2024-04-13T03:31:43.248478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:19:13.039127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562624+0000","last_change":"2024-04-13T03:31:41.848432+0000","last_active":"2024-04-13T03:31:43.562624+0000","last_peered":"2024-04-13T03:31:43.562624+0000","last_clean":"2024-04-13T03:31:43.562624+0000","last_became_active":"2024-04-13T03:31:41.843515+0000","last_became_peered":"2024-04-13T03:31:41.843515+0000","last_unstale":"2024-04-13T03:31:43.562624+0000","last_undegraded":"2024-04-13T03:31:43.562624+0000","last_fullsized":"2024-04-13T03:31:43.562624+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:31:15.512541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.078862+0000","last_change":"2024-04-13T03:31:36.560090+0000","last_active":"2024-04-13T03:32:07.078862+0000","last_peered":"2024-04-13T03:32:07.078862+0000","last_clean":"2024-04-13T03:32:07.078862+0000","last_became_active":"2024-04-13T03:31:36.559770+0000","last_became_peered":"2024-04-13T03:31:36.559770+0000","last_unstale":"2024-04-13T03:32:07.078862+0000","last_undegraded":"2024-04-13T03:32:07.078862+0000","last_fullsized":"2024-04-13T03:32:07.078862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:45:51.715456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835246+0000","last_change":"2024-04-13T03:31:37.593845+0000","last_active":"2024-04-13T03:31:42.835246+0000","last_peered":"2024-04-13T03:31:42.835246+0000","last_clean":"2024-04-13T03:31:42.835246+0000","last_became_active":"2024-04-13T03:31:37.593484+0000","last_became_peered":"2024-04-13T03:31:37.593484+0000","last_unstale":"2024-04-13T03:31:42.835246+0000","last_undegraded":"2024-04-13T03:31:42.835246+0000","last_fullsized":"2024-04-13T03:31:42.835246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:38:05.817746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248747+0000","last_change":"2024-04-13T03:31:41.830949+0000","last_active":"2024-04-13T03:31:43.248747+0000","last_peered":"2024-04-13T03:31:43.248747+0000","last_clean":"2024-04-13T03:31:43.248747+0000","last_became_active":"2024-04-13T03:31:41.830540+0000","last_became_peered":"2024-04-13T03:31:41.830540+0000","last_unstale":"2024-04-13T03:31:43.248747+0000","last_undegraded":"2024-04-13T03:31:43.248747+0000","last_fullsized":"2024-04-13T03:31:43.248747+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:09:00.006158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:08.809274+0000","last_change":"2024-04-13T03:31:36.574773+0000","last_active":"2024-04-13T03:32:08.809274+0000","last_peered":"2024-04-13T03:32:08.809274+0000","last_clean":"2024-04-13T03:32:08.809274+0000","last_became_active":"2024-04-13T03:31:36.574511+0000","last_became_peered":"2024-04-13T03:31:36.574511+0000","last_unstale":"2024-04-13T03:32:08.809274+0000","last_undegraded":"2024-04-13T03:32:08.809274+0000","last_fullsized":"2024-04-13T03:32:08.809274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:26:41.010269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422894+0000","last_change":"2024-04-13T03:31:37.583383+0000","last_active":"2024-04-13T03:31:43.422894+0000","last_peered":"2024-04-13T03:31:43.422894+0000","last_clean":"2024-04-13T03:31:43.422894+0000","last_became_active":"2024-04-13T03:31:37.583026+0000","last_became_peered":"2024-04-13T03:31:37.583026+0000","last_unstale":"2024-04-13T03:31:43.422894+0000","last_undegraded":"2024-04-13T03:31:43.422894+0000","last_fullsized":"2024-04-13T03:31:43.422894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:34.790848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835409+0000","last_change":"2024-04-13T03:31:41.845867+0000","last_active":"2024-04-13T03:31:42.835409+0000","last_peered":"2024-04-13T03:31:42.835409+0000","last_clean":"2024-04-13T03:31:42.835409+0000","last_became_active":"2024-04-13T03:31:41.845408+0000","last_became_peered":"2024-04-13T03:31:41.845408+0000","last_unstale":"2024-04-13T03:31:42.835409+0000","last_undegraded":"2024-04-13T03:31:42.835409+0000","last_fullsized":"2024-04-13T03:31:42.835409+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:10:42.830683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:04.034415+0000","last_change":"2024-04-13T03:31:36.562534+0000","last_active":"2024-04-13T03:32:04.034415+0000","last_peered":"2024-04-13T03:32:04.034415+0000","last_clean":"2024-04-13T03:32:04.034415+0000","last_became_active":"2024-04-13T03:31:36.562189+0000","last_became_peered":"2024-04-13T03:31:36.562189+0000","last_unstale":"2024-04-13T03:32:04.034415+0000","last_undegraded":"2024-04-13T03:32:04.034415+0000","last_fullsized":"2024-04-13T03:32:04.034415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:41:27.504982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562634+0000","last_change":"2024-04-13T03:31:37.588637+0000","last_active":"2024-04-13T03:31:43.562634+0000","last_peered":"2024-04-13T03:31:43.562634+0000","last_clean":"2024-04-13T03:31:43.562634+0000","last_became_active":"2024-04-13T03:31:37.583263+0000","last_became_peered":"2024-04-13T03:31:37.583263+0000","last_unstale":"2024-04-13T03:31:43.562634+0000","last_undegraded":"2024-04-13T03:31:43.562634+0000","last_fullsized":"2024-04-13T03:31:43.562634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:07:40.503310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835767+0000","last_change":"2024-04-13T03:31:41.827302+0000","last_active":"2024-04-13T03:31:42.835767+0000","last_peered":"2024-04-13T03:31:42.835767+0000","last_clean":"2024-04-13T03:31:42.835767+0000","last_became_active":"2024-04-13T03:31:41.826822+0000","last_became_peered":"2024-04-13T03:31:41.826822+0000","last_unstale":"2024-04-13T03:31:42.835767+0000","last_undegraded":"2024-04-13T03:31:42.835767+0000","last_fullsized":"2024-04-13T03:31:42.835767+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:00:16.589319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.072654+0000","last_change":"2024-04-13T03:31:36.582710+0000","last_active":"2024-04-13T03:32:06.072654+0000","last_peered":"2024-04-13T03:32:06.072654+0000","last_clean":"2024-04-13T03:32:06.072654+0000","last_became_active":"2024-04-13T03:31:36.582372+0000","last_became_peered":"2024-04-13T03:31:36.582372+0000","last_unstale":"2024-04-13T03:32:06.072654+0000","last_undegraded":"2024-04-13T03:32:06.072654+0000","last_fullsized":"2024-04-13T03:32:06.072654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:38:58.990256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422774+0000","last_change":"2024-04-13T03:31:37.578345+0000","last_active":"2024-04-13T03:31:43.422774+0000","last_peered":"2024-04-13T03:31:43.422774+0000","last_clean":"2024-04-13T03:31:43.422774+0000","last_became_active":"2024-04-13T03:31:37.578026+0000","last_became_peered":"2024-04-13T03:31:37.578026+0000","last_unstale":"2024-04-13T03:31:43.422774+0000","last_undegraded":"2024-04-13T03:31:43.422774+0000","last_fullsized":"2024-04-13T03:31:43.422774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:59:02.080243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835514+0000","last_change":"2024-04-13T03:31:41.829108+0000","last_active":"2024-04-13T03:31:42.835514+0000","last_peered":"2024-04-13T03:31:42.835514+0000","last_clean":"2024-04-13T03:31:42.835514+0000","last_became_active":"2024-04-13T03:31:41.828603+0000","last_became_peered":"2024-04-13T03:31:41.828603+0000","last_unstale":"2024-04-13T03:31:42.835514+0000","last_undegraded":"2024-04-13T03:31:42.835514+0000","last_fullsized":"2024-04-13T03:31:42.835514+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:06:32.753928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'26","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:08.814345+0000","last_change":"2024-04-13T03:31:36.562447+0000","last_active":"2024-04-13T03:32:08.814345+0000","last_peered":"2024-04-13T03:32:08.814345+0000","last_clean":"2024-04-13T03:32:08.814345+0000","last_became_active":"2024-04-13T03:31:36.562094+0000","last_became_peered":"2024-04-13T03:31:36.562094+0000","last_unstale":"2024-04-13T03:32:08.814345+0000","last_undegraded":"2024-04-13T03:32:08.814345+0000","last_fullsized":"2024-04-13T03:32:08.814345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:04:26.847519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41906,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":44,"num_write":28,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563024+0000","last_change":"2024-04-13T03:31:37.580515+0000","last_active":"2024-04-13T03:31:43.563024+0000","last_peered":"2024-04-13T03:31:43.563024+0000","last_clean":"2024-04-13T03:31:43.563024+0000","last_became_active":"2024-04-13T03:31:37.580157+0000","last_became_peered":"2024-04-13T03:31:37.580157+0000","last_unstale":"2024-04-13T03:31:43.563024+0000","last_undegraded":"2024-04-13T03:31:43.563024+0000","last_fullsized":"2024-04-13T03:31:43.563024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:17:17.843273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834856+0000","last_change":"2024-04-13T03:31:41.833555+0000","last_active":"2024-04-13T03:31:42.834856+0000","last_peered":"2024-04-13T03:31:42.834856+0000","last_clean":"2024-04-13T03:31:42.834856+0000","last_became_active":"2024-04-13T03:31:41.833134+0000","last_became_peered":"2024-04-13T03:31:41.833134+0000","last_unstale":"2024-04-13T03:31:42.834856+0000","last_undegraded":"2024-04-13T03:31:42.834856+0000","last_fullsized":"2024-04-13T03:31:42.834856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:22:58.311184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.048771+0000","last_change":"2024-04-13T03:31:36.591127+0000","last_active":"2024-04-13T03:32:05.048771+0000","last_peered":"2024-04-13T03:32:05.048771+0000","last_clean":"2024-04-13T03:32:05.048771+0000","last_became_active":"2024-04-13T03:31:36.590819+0000","last_became_peered":"2024-04-13T03:31:36.590819+0000","last_unstale":"2024-04-13T03:32:05.048771+0000","last_undegraded":"2024-04-13T03:32:05.048771+0000","last_fullsized":"2024-04-13T03:32:05.048771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:54:51.637340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846695+0000","last_change":"2024-04-13T03:31:37.583147+0000","last_active":"2024-04-13T03:31:42.846695+0000","last_peered":"2024-04-13T03:31:42.846695+0000","last_clean":"2024-04-13T03:31:42.846695+0000","last_became_active":"2024-04-13T03:31:37.582781+0000","last_became_peered":"2024-04-13T03:31:37.582781+0000","last_unstale":"2024-04-13T03:31:42.846695+0000","last_undegraded":"2024-04-13T03:31:42.846695+0000","last_fullsized":"2024-04-13T03:31:42.846695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:47:36.879512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563488+0000","last_change":"2024-04-13T03:31:41.833559+0000","last_active":"2024-04-13T03:31:43.563488+0000","last_peered":"2024-04-13T03:31:43.563488+0000","last_clean":"2024-04-13T03:31:43.563488+0000","last_became_active":"2024-04-13T03:31:41.833067+0000","last_became_peered":"2024-04-13T03:31:41.833067+0000","last_unstale":"2024-04-13T03:31:43.563488+0000","last_undegraded":"2024-04-13T03:31:43.563488+0000","last_fullsized":"2024-04-13T03:31:43.563488+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:56:09.027293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.084769+0000","last_change":"2024-04-13T03:31:36.597136+0000","last_active":"2024-04-13T03:32:07.084769+0000","last_peered":"2024-04-13T03:32:07.084769+0000","last_clean":"2024-04-13T03:32:07.084769+0000","last_became_active":"2024-04-13T03:31:36.596789+0000","last_became_peered":"2024-04-13T03:31:36.596789+0000","last_unstale":"2024-04-13T03:32:07.084769+0000","last_undegraded":"2024-04-13T03:32:07.084769+0000","last_fullsized":"2024-04-13T03:32:07.084769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:39:28.371669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562806+0000","last_change":"2024-04-13T03:31:37.584424+0000","last_active":"2024-04-13T03:31:43.562806+0000","last_peered":"2024-04-13T03:31:43.562806+0000","last_clean":"2024-04-13T03:31:43.562806+0000","last_became_active":"2024-04-13T03:31:37.584125+0000","last_became_peered":"2024-04-13T03:31:37.584125+0000","last_unstale":"2024-04-13T03:31:43.562806+0000","last_undegraded":"2024-04-13T03:31:43.562806+0000","last_fullsized":"2024-04-13T03:31:43.562806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:07:53.625486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836253+0000","last_change":"2024-04-13T03:31:41.852755+0000","last_active":"2024-04-13T03:31:42.836253+0000","last_peered":"2024-04-13T03:31:42.836253+0000","last_clean":"2024-04-13T03:31:42.836253+0000","last_became_active":"2024-04-13T03:31:41.851975+0000","last_became_peered":"2024-04-13T03:31:41.851975+0000","last_unstale":"2024-04-13T03:31:42.836253+0000","last_undegraded":"2024-04-13T03:31:42.836253+0000","last_fullsized":"2024-04-13T03:31:42.836253+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:19:40.982525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.064509+0000","last_change":"2024-04-13T03:31:36.559351+0000","last_active":"2024-04-13T03:32:06.064509+0000","last_peered":"2024-04-13T03:32:06.064509+0000","last_clean":"2024-04-13T03:32:06.064509+0000","last_became_active":"2024-04-13T03:31:36.559011+0000","last_became_peered":"2024-04-13T03:31:36.559011+0000","last_unstale":"2024-04-13T03:32:06.064509+0000","last_undegraded":"2024-04-13T03:32:06.064509+0000","last_fullsized":"2024-04-13T03:32:06.064509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:08:08.983088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562255+0000","last_change":"2024-04-13T03:31:37.580884+0000","last_active":"2024-04-13T03:31:43.562255+0000","last_peered":"2024-04-13T03:31:43.562255+0000","last_clean":"2024-04-13T03:31:43.562255+0000","last_became_active":"2024-04-13T03:31:37.580568+0000","last_became_peered":"2024-04-13T03:31:37.580568+0000","last_unstale":"2024-04-13T03:31:43.562255+0000","last_undegraded":"2024-04-13T03:31:43.562255+0000","last_fullsized":"2024-04-13T03:31:43.562255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:42:41.484614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.847286+0000","last_change":"2024-04-13T03:31:41.840912+0000","last_active":"2024-04-13T03:31:42.847286+0000","last_peered":"2024-04-13T03:31:42.847286+0000","last_clean":"2024-04-13T03:31:42.847286+0000","last_became_active":"2024-04-13T03:31:41.840129+0000","last_became_peered":"2024-04-13T03:31:41.840129+0000","last_unstale":"2024-04-13T03:31:42.847286+0000","last_undegraded":"2024-04-13T03:31:42.847286+0000","last_fullsized":"2024-04-13T03:31:42.847286+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:45:33.716345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.046113+0000","last_change":"2024-04-13T03:31:36.580230+0000","last_active":"2024-04-13T03:32:05.046113+0000","last_peered":"2024-04-13T03:32:05.046113+0000","last_clean":"2024-04-13T03:32:05.046113+0000","last_became_active":"2024-04-13T03:31:36.579874+0000","last_became_peered":"2024-04-13T03:31:36.579874+0000","last_unstale":"2024-04-13T03:32:05.046113+0000","last_undegraded":"2024-04-13T03:32:05.046113+0000","last_fullsized":"2024-04-13T03:32:05.046113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:30:17.154748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834014+0000","last_change":"2024-04-13T03:31:37.581937+0000","last_active":"2024-04-13T03:31:42.834014+0000","last_peered":"2024-04-13T03:31:42.834014+0000","last_clean":"2024-04-13T03:31:42.834014+0000","last_became_active":"2024-04-13T03:31:37.581547+0000","last_became_peered":"2024-04-13T03:31:37.581547+0000","last_unstale":"2024-04-13T03:31:42.834014+0000","last_undegraded":"2024-04-13T03:31:42.834014+0000","last_fullsized":"2024-04-13T03:31:42.834014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:19:28.600996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834086+0000","last_change":"2024-04-13T03:31:41.842887+0000","last_active":"2024-04-13T03:31:42.834086+0000","last_peered":"2024-04-13T03:31:42.834086+0000","last_clean":"2024-04-13T03:31:42.834086+0000","last_became_active":"2024-04-13T03:31:41.842537+0000","last_became_peered":"2024-04-13T03:31:41.842537+0000","last_unstale":"2024-04-13T03:31:42.834086+0000","last_undegraded":"2024-04-13T03:31:42.834086+0000","last_fullsized":"2024-04-13T03:31:42.834086+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:44:38.930759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563300+0000","last_change":"2024-04-13T03:31:36.561347+0000","last_active":"2024-04-13T03:31:43.563300+0000","last_peered":"2024-04-13T03:31:43.563300+0000","last_clean":"2024-04-13T03:31:43.563300+0000","last_became_active":"2024-04-13T03:31:36.560969+0000","last_became_peered":"2024-04-13T03:31:36.560969+0000","last_unstale":"2024-04-13T03:31:43.563300+0000","last_undegraded":"2024-04-13T03:31:43.563300+0000","last_fullsized":"2024-04-13T03:31:43.563300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:59:38.825930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563519+0000","last_change":"2024-04-13T03:31:37.587599+0000","last_active":"2024-04-13T03:31:43.563519+0000","last_peered":"2024-04-13T03:31:43.563519+0000","last_clean":"2024-04-13T03:31:43.563519+0000","last_became_active":"2024-04-13T03:31:37.587257+0000","last_became_peered":"2024-04-13T03:31:37.587257+0000","last_unstale":"2024-04-13T03:31:43.563519+0000","last_undegraded":"2024-04-13T03:31:43.563519+0000","last_fullsized":"2024-04-13T03:31:43.563519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:39:57.956174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563296+0000","last_change":"2024-04-13T03:31:41.831134+0000","last_active":"2024-04-13T03:31:43.563296+0000","last_peered":"2024-04-13T03:31:43.563296+0000","last_clean":"2024-04-13T03:31:43.563296+0000","last_became_active":"2024-04-13T03:31:41.830671+0000","last_became_peered":"2024-04-13T03:31:41.830671+0000","last_unstale":"2024-04-13T03:31:43.563296+0000","last_undegraded":"2024-04-13T03:31:43.563296+0000","last_fullsized":"2024-04-13T03:31:43.563296+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:50:59.527951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836188+0000","last_change":"2024-04-13T03:31:36.586008+0000","last_active":"2024-04-13T03:31:42.836188+0000","last_peered":"2024-04-13T03:31:42.836188+0000","last_clean":"2024-04-13T03:31:42.836188+0000","last_became_active":"2024-04-13T03:31:36.583937+0000","last_became_peered":"2024-04-13T03:31:36.583937+0000","last_unstale":"2024-04-13T03:31:42.836188+0000","last_undegraded":"2024-04-13T03:31:42.836188+0000","last_fullsized":"2024-04-13T03:31:42.836188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:17:06.194397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562737+0000","last_change":"2024-04-13T03:31:37.586185+0000","last_active":"2024-04-13T03:31:43.562737+0000","last_peered":"2024-04-13T03:31:43.562737+0000","last_clean":"2024-04-13T03:31:43.562737+0000","last_became_active":"2024-04-13T03:31:37.585509+0000","last_became_peered":"2024-04-13T03:31:37.585509+0000","last_unstale":"2024-04-13T03:31:43.562737+0000","last_undegraded":"2024-04-13T03:31:43.562737+0000","last_fullsized":"2024-04-13T03:31:43.562737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:35:41.064317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846911+0000","last_change":"2024-04-13T03:31:41.847376+0000","last_active":"2024-04-13T03:31:42.846911+0000","last_peered":"2024-04-13T03:31:42.846911+0000","last_clean":"2024-04-13T03:31:42.846911+0000","last_became_active":"2024-04-13T03:31:41.846724+0000","last_became_peered":"2024-04-13T03:31:41.846724+0000","last_unstale":"2024-04-13T03:31:42.846911+0000","last_undegraded":"2024-04-13T03:31:42.846911+0000","last_fullsized":"2024-04-13T03:31:42.846911+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:26:14.046875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562510+0000","last_change":"2024-04-13T03:31:36.582776+0000","last_active":"2024-04-13T03:31:43.562510+0000","last_peered":"2024-04-13T03:31:43.562510+0000","last_clean":"2024-04-13T03:31:43.562510+0000","last_became_active":"2024-04-13T03:31:36.582405+0000","last_became_peered":"2024-04-13T03:31:36.582405+0000","last_unstale":"2024-04-13T03:31:43.562510+0000","last_undegraded":"2024-04-13T03:31:43.562510+0000","last_fullsized":"2024-04-13T03:31:43.562510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:18:56.624442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562925+0000","last_change":"2024-04-13T03:31:37.580552+0000","last_active":"2024-04-13T03:31:43.562925+0000","last_peered":"2024-04-13T03:31:43.562925+0000","last_clean":"2024-04-13T03:31:43.562925+0000","last_became_active":"2024-04-13T03:31:37.576390+0000","last_became_peered":"2024-04-13T03:31:37.576390+0000","last_unstale":"2024-04-13T03:31:43.562925+0000","last_undegraded":"2024-04-13T03:31:43.562925+0000","last_fullsized":"2024-04-13T03:31:43.562925+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:50:48.328048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.847240+0000","last_change":"2024-04-13T03:31:41.849818+0000","last_active":"2024-04-13T03:31:42.847240+0000","last_peered":"2024-04-13T03:31:42.847240+0000","last_clean":"2024-04-13T03:31:42.847240+0000","last_became_active":"2024-04-13T03:31:41.848846+0000","last_became_peered":"2024-04-13T03:31:41.848846+0000","last_unstale":"2024-04-13T03:31:42.847240+0000","last_undegraded":"2024-04-13T03:31:42.847240+0000","last_fullsized":"2024-04-13T03:31:42.847240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:55:16.764935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834185+0000","last_change":"2024-04-13T03:31:37.581723+0000","last_active":"2024-04-13T03:31:42.834185+0000","last_peered":"2024-04-13T03:31:42.834185+0000","last_clean":"2024-04-13T03:31:42.834185+0000","last_became_active":"2024-04-13T03:31:37.578857+0000","last_became_peered":"2024-04-13T03:31:37.578857+0000","last_unstale":"2024-04-13T03:31:42.834185+0000","last_undegraded":"2024-04-13T03:31:42.834185+0000","last_fullsized":"2024-04-13T03:31:42.834185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:47:34.447256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.807240+0000","last_change":"2024-04-13T03:31:36.592275+0000","last_active":"2024-04-13T03:32:07.807240+0000","last_peered":"2024-04-13T03:32:07.807240+0000","last_clean":"2024-04-13T03:32:07.807240+0000","last_became_active":"2024-04-13T03:31:36.591905+0000","last_became_peered":"2024-04-13T03:31:36.591905+0000","last_unstale":"2024-04-13T03:32:07.807240+0000","last_undegraded":"2024-04-13T03:32:07.807240+0000","last_fullsized":"2024-04-13T03:32:07.807240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563104+0000","last_change":"2024-04-13T03:31:41.835412+0000","last_active":"2024-04-13T03:31:43.563104+0000","last_peered":"2024-04-13T03:31:43.563104+0000","last_clean":"2024-04-13T03:31:43.563104+0000","last_became_active":"2024-04-13T03:31:41.834978+0000","last_became_peered":"2024-04-13T03:31:41.834978+0000","last_unstale":"2024-04-13T03:31:43.563104+0000","last_undegraded":"2024-04-13T03:31:43.563104+0000","last_fullsized":"2024-04-13T03:31:43.563104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:45:52.037452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248672+0000","last_change":"2024-04-13T03:31:37.585501+0000","last_active":"2024-04-13T03:31:43.248672+0000","last_peered":"2024-04-13T03:31:43.248672+0000","last_clean":"2024-04-13T03:31:43.248672+0000","last_became_active":"2024-04-13T03:31:37.584965+0000","last_became_peered":"2024-04-13T03:31:37.584965+0000","last_unstale":"2024-04-13T03:31:43.248672+0000","last_undegraded":"2024-04-13T03:31:43.248672+0000","last_fullsized":"2024-04-13T03:31:43.248672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:44.017270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.074766+0000","last_change":"2024-04-13T03:31:36.580058+0000","last_active":"2024-04-13T03:32:06.074766+0000","last_peered":"2024-04-13T03:32:06.074766+0000","last_clean":"2024-04-13T03:32:06.074766+0000","last_became_active":"2024-04-13T03:31:36.579777+0000","last_became_peered":"2024-04-13T03:31:36.579777+0000","last_unstale":"2024-04-13T03:32:06.074766+0000","last_undegraded":"2024-04-13T03:32:06.074766+0000","last_fullsized":"2024-04-13T03:32:06.074766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:27:50.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562085+0000","last_change":"2024-04-13T03:30:17.781014+0000","last_active":"2024-04-13T03:31:43.562085+0000","last_peered":"2024-04-13T03:31:43.562085+0000","last_clean":"2024-04-13T03:31:43.562085+0000","last_became_active":"2024-04-13T03:30:17.780417+0000","last_became_peered":"2024-04-13T03:30:17.780417+0000","last_unstale":"2024-04-13T03:31:43.562085+0000","last_undegraded":"2024-04-13T03:31:43.562085+0000","last_fullsized":"2024-04-13T03:31:43.562085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:26:22.174408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:26:22.174408+0000","last_clean_scrub_stamp":"2024-04-13T03:26:22.174408+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:52:50.418399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834867+0000","last_change":"2024-04-13T03:31:41.837973+0000","last_active":"2024-04-13T03:31:42.834867+0000","last_peered":"2024-04-13T03:31:42.834867+0000","last_clean":"2024-04-13T03:31:42.834867+0000","last_became_active":"2024-04-13T03:31:41.837562+0000","last_became_peered":"2024-04-13T03:31:41.837562+0000","last_unstale":"2024-04-13T03:31:42.834867+0000","last_undegraded":"2024-04-13T03:31:42.834867+0000","last_fullsized":"2024-04-13T03:31:42.834867+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:05:09.900193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846410+0000","last_change":"2024-04-13T03:31:37.584279+0000","last_active":"2024-04-13T03:31:42.846410+0000","last_peered":"2024-04-13T03:31:42.846410+0000","last_clean":"2024-04-13T03:31:42.846410+0000","last_became_active":"2024-04-13T03:31:37.583891+0000","last_became_peered":"2024-04-13T03:31:37.583891+0000","last_unstale":"2024-04-13T03:31:42.846410+0000","last_undegraded":"2024-04-13T03:31:42.846410+0000","last_fullsized":"2024-04-13T03:31:42.846410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:56:21.446525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.043502+0000","last_change":"2024-04-13T03:31:36.561264+0000","last_active":"2024-04-13T03:32:05.043502+0000","last_peered":"2024-04-13T03:32:05.043502+0000","last_clean":"2024-04-13T03:32:05.043502+0000","last_became_active":"2024-04-13T03:31:36.560921+0000","last_became_peered":"2024-04-13T03:31:36.560921+0000","last_unstale":"2024-04-13T03:32:05.043502+0000","last_undegraded":"2024-04-13T03:32:05.043502+0000","last_fullsized":"2024-04-13T03:32:05.043502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:10.420841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563558+0000","last_change":"2024-04-13T03:31:41.833350+0000","last_active":"2024-04-13T03:31:43.563558+0000","last_peered":"2024-04-13T03:31:43.563558+0000","last_clean":"2024-04-13T03:31:43.563558+0000","last_became_active":"2024-04-13T03:31:41.832865+0000","last_became_peered":"2024-04-13T03:31:41.832865+0000","last_unstale":"2024-04-13T03:31:43.563558+0000","last_undegraded":"2024-04-13T03:31:43.563558+0000","last_fullsized":"2024-04-13T03:31:43.563558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:23:31.872562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563398+0000","last_change":"2024-04-13T03:31:36.587435+0000","last_active":"2024-04-13T03:31:43.563398+0000","last_peered":"2024-04-13T03:31:43.563398+0000","last_clean":"2024-04-13T03:31:43.563398+0000","last_became_active":"2024-04-13T03:31:36.587187+0000","last_became_peered":"2024-04-13T03:31:36.587187+0000","last_unstale":"2024-04-13T03:31:43.563398+0000","last_undegraded":"2024-04-13T03:31:43.563398+0000","last_fullsized":"2024-04-13T03:31:43.563398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:06:18.665638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846964+0000","last_change":"2024-04-13T03:31:37.584429+0000","last_active":"2024-04-13T03:31:42.846964+0000","last_peered":"2024-04-13T03:31:42.846964+0000","last_clean":"2024-04-13T03:31:42.846964+0000","last_became_active":"2024-04-13T03:31:37.583811+0000","last_became_peered":"2024-04-13T03:31:37.583811+0000","last_unstale":"2024-04-13T03:31:42.846964+0000","last_undegraded":"2024-04-13T03:31:42.846964+0000","last_fullsized":"2024-04-13T03:31:42.846964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:52:20.227001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248493+0000","last_change":"2024-04-13T03:31:41.846065+0000","last_active":"2024-04-13T03:31:43.248493+0000","last_peered":"2024-04-13T03:31:43.248493+0000","last_clean":"2024-04-13T03:31:43.248493+0000","last_became_active":"2024-04-13T03:31:41.844422+0000","last_became_peered":"2024-04-13T03:31:41.844422+0000","last_unstale":"2024-04-13T03:31:43.248493+0000","last_undegraded":"2024-04-13T03:31:43.248493+0000","last_fullsized":"2024-04-13T03:31:43.248493+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:51:41.152066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.051887+0000","last_change":"2024-04-13T03:31:36.598323+0000","last_active":"2024-04-13T03:32:05.051887+0000","last_peered":"2024-04-13T03:32:05.051887+0000","last_clean":"2024-04-13T03:32:05.051887+0000","last_became_active":"2024-04-13T03:31:36.597412+0000","last_became_peered":"2024-04-13T03:31:36.597412+0000","last_unstale":"2024-04-13T03:32:05.051887+0000","last_undegraded":"2024-04-13T03:32:05.051887+0000","last_fullsized":"2024-04-13T03:32:05.051887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:19:19.203273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.249939+0000","last_change":"2024-04-13T03:31:37.591599+0000","last_active":"2024-04-13T03:31:43.249939+0000","last_peered":"2024-04-13T03:31:43.249939+0000","last_clean":"2024-04-13T03:31:43.249939+0000","last_became_active":"2024-04-13T03:31:37.591243+0000","last_became_peered":"2024-04-13T03:31:37.591243+0000","last_unstale":"2024-04-13T03:31:43.249939+0000","last_undegraded":"2024-04-13T03:31:43.249939+0000","last_fullsized":"2024-04-13T03:31:43.249939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:11:35.460721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562028+0000","last_change":"2024-04-13T03:31:41.821604+0000","last_active":"2024-04-13T03:31:43.562028+0000","last_peered":"2024-04-13T03:31:43.562028+0000","last_clean":"2024-04-13T03:31:43.562028+0000","last_became_active":"2024-04-13T03:31:41.820804+0000","last_became_peered":"2024-04-13T03:31:41.820804+0000","last_unstale":"2024-04-13T03:31:43.562028+0000","last_undegraded":"2024-04-13T03:31:43.562028+0000","last_fullsized":"2024-04-13T03:31:43.562028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:10:45.850099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:04.039988+0000","last_change":"2024-04-13T03:31:36.582624+0000","last_active":"2024-04-13T03:32:04.039988+0000","last_peered":"2024-04-13T03:32:04.039988+0000","last_clean":"2024-04-13T03:32:04.039988+0000","last_became_active":"2024-04-13T03:31:36.582260+0000","last_became_peered":"2024-04-13T03:31:36.582260+0000","last_unstale":"2024-04-13T03:32:04.039988+0000","last_undegraded":"2024-04-13T03:32:04.039988+0000","last_fullsized":"2024-04-13T03:32:04.039988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:15.637016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562761+0000","last_change":"2024-04-13T03:31:37.591286+0000","last_active":"2024-04-13T03:31:43.562761+0000","last_peered":"2024-04-13T03:31:43.562761+0000","last_clean":"2024-04-13T03:31:43.562761+0000","last_became_active":"2024-04-13T03:31:37.590891+0000","last_became_peered":"2024-04-13T03:31:37.590891+0000","last_unstale":"2024-04-13T03:31:43.562761+0000","last_undegraded":"2024-04-13T03:31:43.562761+0000","last_fullsized":"2024-04-13T03:31:43.562761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:53:54.696082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835922+0000","last_change":"2024-04-13T03:31:41.830499+0000","last_active":"2024-04-13T03:31:42.835922+0000","last_peered":"2024-04-13T03:31:42.835922+0000","last_clean":"2024-04-13T03:31:42.835922+0000","last_became_active":"2024-04-13T03:31:41.830067+0000","last_became_peered":"2024-04-13T03:31:41.830067+0000","last_unstale":"2024-04-13T03:31:42.835922+0000","last_undegraded":"2024-04-13T03:31:42.835922+0000","last_fullsized":"2024-04-13T03:31:42.835922+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:22.635885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:04.037302+0000","last_change":"2024-04-13T03:31:36.579307+0000","last_active":"2024-04-13T03:32:04.037302+0000","last_peered":"2024-04-13T03:32:04.037302+0000","last_clean":"2024-04-13T03:32:04.037302+0000","last_became_active":"2024-04-13T03:31:36.579068+0000","last_became_peered":"2024-04-13T03:31:36.579068+0000","last_unstale":"2024-04-13T03:32:04.037302+0000","last_undegraded":"2024-04-13T03:32:04.037302+0000","last_fullsized":"2024-04-13T03:32:04.037302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:49:09.187151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562459+0000","last_change":"2024-04-13T03:31:37.575749+0000","last_active":"2024-04-13T03:31:43.562459+0000","last_peered":"2024-04-13T03:31:43.562459+0000","last_clean":"2024-04-13T03:31:43.562459+0000","last_became_active":"2024-04-13T03:31:37.574152+0000","last_became_peered":"2024-04-13T03:31:37.574152+0000","last_unstale":"2024-04-13T03:31:43.562459+0000","last_undegraded":"2024-04-13T03:31:43.562459+0000","last_fullsized":"2024-04-13T03:31:43.562459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:38:15.710978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836676+0000","last_change":"2024-04-13T03:31:41.836777+0000","last_active":"2024-04-13T03:31:42.836676+0000","last_peered":"2024-04-13T03:31:42.836676+0000","last_clean":"2024-04-13T03:31:42.836676+0000","last_became_active":"2024-04-13T03:31:41.835877+0000","last_became_peered":"2024-04-13T03:31:41.835877+0000","last_unstale":"2024-04-13T03:31:42.836676+0000","last_undegraded":"2024-04-13T03:31:42.836676+0000","last_fullsized":"2024-04-13T03:31:42.836676+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:46:44.174095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.084306+0000","last_change":"2024-04-13T03:31:36.587981+0000","last_active":"2024-04-13T03:32:07.084306+0000","last_peered":"2024-04-13T03:32:07.084306+0000","last_clean":"2024-04-13T03:32:07.084306+0000","last_became_active":"2024-04-13T03:31:36.586013+0000","last_became_peered":"2024-04-13T03:31:36.586013+0000","last_unstale":"2024-04-13T03:32:07.084306+0000","last_undegraded":"2024-04-13T03:32:07.084306+0000","last_fullsized":"2024-04-13T03:32:07.084306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:16:27.075322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562682+0000","last_change":"2024-04-13T03:31:37.594185+0000","last_active":"2024-04-13T03:31:43.562682+0000","last_peered":"2024-04-13T03:31:43.562682+0000","last_clean":"2024-04-13T03:31:43.562682+0000","last_became_active":"2024-04-13T03:31:37.593791+0000","last_became_peered":"2024-04-13T03:31:37.593791+0000","last_unstale":"2024-04-13T03:31:43.562682+0000","last_undegraded":"2024-04-13T03:31:43.562682+0000","last_fullsized":"2024-04-13T03:31:43.562682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:43:32.035371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.835783+0000","last_change":"2024-04-13T03:31:41.830392+0000","last_active":"2024-04-13T03:31:42.835783+0000","last_peered":"2024-04-13T03:31:42.835783+0000","last_clean":"2024-04-13T03:31:42.835783+0000","last_became_active":"2024-04-13T03:31:41.829967+0000","last_became_peered":"2024-04-13T03:31:41.829967+0000","last_unstale":"2024-04-13T03:31:42.835783+0000","last_undegraded":"2024-04-13T03:31:42.835783+0000","last_fullsized":"2024-04-13T03:31:42.835783+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:43:25.326951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:05.031299+0000","last_change":"2024-04-13T03:31:36.597099+0000","last_active":"2024-04-13T03:32:05.031299+0000","last_peered":"2024-04-13T03:32:05.031299+0000","last_clean":"2024-04-13T03:32:05.031299+0000","last_became_active":"2024-04-13T03:31:36.596714+0000","last_became_peered":"2024-04-13T03:31:36.596714+0000","last_unstale":"2024-04-13T03:32:05.031299+0000","last_undegraded":"2024-04-13T03:32:05.031299+0000","last_fullsized":"2024-04-13T03:32:05.031299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:42:31.477730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.249076+0000","last_change":"2024-04-13T03:31:37.587038+0000","last_active":"2024-04-13T03:31:43.249076+0000","last_peered":"2024-04-13T03:31:43.249076+0000","last_clean":"2024-04-13T03:31:43.249076+0000","last_became_active":"2024-04-13T03:31:37.586618+0000","last_became_peered":"2024-04-13T03:31:37.586618+0000","last_unstale":"2024-04-13T03:31:43.249076+0000","last_undegraded":"2024-04-13T03:31:43.249076+0000","last_fullsized":"2024-04-13T03:31:43.249076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:57:33.868529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.836017+0000","last_change":"2024-04-13T03:31:41.833148+0000","last_active":"2024-04-13T03:31:42.836017+0000","last_peered":"2024-04-13T03:31:42.836017+0000","last_clean":"2024-04-13T03:31:42.836017+0000","last_became_active":"2024-04-13T03:31:41.832714+0000","last_became_peered":"2024-04-13T03:31:41.832714+0000","last_unstale":"2024-04-13T03:31:42.836017+0000","last_undegraded":"2024-04-13T03:31:42.836017+0000","last_fullsized":"2024-04-13T03:31:42.836017+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:12:07.316488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:04.033925+0000","last_change":"2024-04-13T03:31:36.564887+0000","last_active":"2024-04-13T03:32:04.033925+0000","last_peered":"2024-04-13T03:32:04.033925+0000","last_clean":"2024-04-13T03:32:04.033925+0000","last_became_active":"2024-04-13T03:31:36.564438+0000","last_became_peered":"2024-04-13T03:31:36.564438+0000","last_unstale":"2024-04-13T03:32:04.033925+0000","last_undegraded":"2024-04-13T03:32:04.033925+0000","last_fullsized":"2024-04-13T03:32:04.033925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:19.310322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562562+0000","last_change":"2024-04-13T03:31:37.584358+0000","last_active":"2024-04-13T03:31:43.562562+0000","last_peered":"2024-04-13T03:31:43.562562+0000","last_clean":"2024-04-13T03:31:43.562562+0000","last_became_active":"2024-04-13T03:31:37.584005+0000","last_became_peered":"2024-04-13T03:31:37.584005+0000","last_unstale":"2024-04-13T03:31:43.562562+0000","last_undegraded":"2024-04-13T03:31:43.562562+0000","last_fullsized":"2024-04-13T03:31:43.562562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:07:21.465290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422621+0000","last_change":"2024-04-13T03:31:41.833451+0000","last_active":"2024-04-13T03:31:43.422621+0000","last_peered":"2024-04-13T03:31:43.422621+0000","last_clean":"2024-04-13T03:31:43.422621+0000","last_became_active":"2024-04-13T03:31:41.832734+0000","last_became_peered":"2024-04-13T03:31:41.832734+0000","last_unstale":"2024-04-13T03:31:43.422621+0000","last_undegraded":"2024-04-13T03:31:43.422621+0000","last_fullsized":"2024-04-13T03:31:43.422621+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:49:48.848152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:06.070843+0000","last_change":"2024-04-13T03:31:36.581697+0000","last_active":"2024-04-13T03:32:06.070843+0000","last_peered":"2024-04-13T03:32:06.070843+0000","last_clean":"2024-04-13T03:32:06.070843+0000","last_became_active":"2024-04-13T03:31:36.581345+0000","last_became_peered":"2024-04-13T03:31:36.581345+0000","last_unstale":"2024-04-13T03:32:06.070843+0000","last_undegraded":"2024-04-13T03:32:06.070843+0000","last_fullsized":"2024-04-13T03:32:06.070843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:29:53.192770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563234+0000","last_change":"2024-04-13T03:31:37.571879+0000","last_active":"2024-04-13T03:31:43.563234+0000","last_peered":"2024-04-13T03:31:43.563234+0000","last_clean":"2024-04-13T03:31:43.563234+0000","last_became_active":"2024-04-13T03:31:37.571618+0000","last_became_peered":"2024-04-13T03:31:37.571618+0000","last_unstale":"2024-04-13T03:31:43.563234+0000","last_undegraded":"2024-04-13T03:31:43.563234+0000","last_fullsized":"2024-04-13T03:31:43.563234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:42.920388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562312+0000","last_change":"2024-04-13T03:31:41.851871+0000","last_active":"2024-04-13T03:31:43.562312+0000","last_peered":"2024-04-13T03:31:43.562312+0000","last_clean":"2024-04-13T03:31:43.562312+0000","last_became_active":"2024-04-13T03:31:41.851045+0000","last_became_peered":"2024-04-13T03:31:41.851045+0000","last_unstale":"2024-04-13T03:31:43.562312+0000","last_undegraded":"2024-04-13T03:31:43.562312+0000","last_fullsized":"2024-04-13T03:31:43.562312+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:35:41.937723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.083523+0000","last_change":"2024-04-13T03:31:36.586110+0000","last_active":"2024-04-13T03:32:07.083523+0000","last_peered":"2024-04-13T03:32:07.083523+0000","last_clean":"2024-04-13T03:32:07.083523+0000","last_became_active":"2024-04-13T03:31:36.585832+0000","last_became_peered":"2024-04-13T03:31:36.585832+0000","last_unstale":"2024-04-13T03:32:07.083523+0000","last_undegraded":"2024-04-13T03:32:07.083523+0000","last_fullsized":"2024-04-13T03:32:07.083523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:00:57.825760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562919+0000","last_change":"2024-04-13T03:31:37.580004+0000","last_active":"2024-04-13T03:31:43.562919+0000","last_peered":"2024-04-13T03:31:43.562919+0000","last_clean":"2024-04-13T03:31:43.562919+0000","last_became_active":"2024-04-13T03:31:37.579634+0000","last_became_peered":"2024-04-13T03:31:37.579634+0000","last_unstale":"2024-04-13T03:31:43.562919+0000","last_undegraded":"2024-04-13T03:31:43.562919+0000","last_fullsized":"2024-04-13T03:31:43.562919+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:39:59.872588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.833865+0000","last_change":"2024-04-13T03:31:41.825456+0000","last_active":"2024-04-13T03:31:42.833865+0000","last_peered":"2024-04-13T03:31:42.833865+0000","last_clean":"2024-04-13T03:31:42.833865+0000","last_became_active":"2024-04-13T03:31:41.824929+0000","last_became_peered":"2024-04-13T03:31:41.824929+0000","last_unstale":"2024-04-13T03:31:42.833865+0000","last_undegraded":"2024-04-13T03:31:42.833865+0000","last_fullsized":"2024-04-13T03:31:42.833865+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:17:39.405147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:32:07.084252+0000","last_change":"2024-04-13T03:31:36.595009+0000","last_active":"2024-04-13T03:32:07.084252+0000","last_peered":"2024-04-13T03:32:07.084252+0000","last_clean":"2024-04-13T03:32:07.084252+0000","last_became_active":"2024-04-13T03:31:36.594314+0000","last_became_peered":"2024-04-13T03:31:36.594314+0000","last_unstale":"2024-04-13T03:32:07.084252+0000","last_undegraded":"2024-04-13T03:32:07.084252+0000","last_fullsized":"2024-04-13T03:32:07.084252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:42:33.570088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562901+0000","last_change":"2024-04-13T03:31:37.579899+0000","last_active":"2024-04-13T03:31:43.562901+0000","last_peered":"2024-04-13T03:31:43.562901+0000","last_clean":"2024-04-13T03:31:43.562901+0000","last_became_active":"2024-04-13T03:31:37.579345+0000","last_became_peered":"2024-04-13T03:31:37.579345+0000","last_unstale":"2024-04-13T03:31:43.562901+0000","last_undegraded":"2024-04-13T03:31:43.562901+0000","last_fullsized":"2024-04-13T03:31:43.562901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:06:55.646866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562802+0000","last_change":"2024-04-13T03:31:37.586373+0000","last_active":"2024-04-13T03:31:43.562802+0000","last_peered":"2024-04-13T03:31:43.562802+0000","last_clean":"2024-04-13T03:31:43.562802+0000","last_became_active":"2024-04-13T03:31:37.586061+0000","last_became_peered":"2024-04-13T03:31:37.586061+0000","last_unstale":"2024-04-13T03:31:43.562802+0000","last_undegraded":"2024-04-13T03:31:43.562802+0000","last_fullsized":"2024-04-13T03:31:43.562802+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:58:52.487187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562380+0000","last_change":"2024-04-13T03:31:37.581375+0000","last_active":"2024-04-13T03:31:43.562380+0000","last_peered":"2024-04-13T03:31:43.562380+0000","last_clean":"2024-04-13T03:31:43.562380+0000","last_became_active":"2024-04-13T03:31:37.580802+0000","last_became_peered":"2024-04-13T03:31:37.580802+0000","last_unstale":"2024-04-13T03:31:43.562380+0000","last_undegraded":"2024-04-13T03:31:43.562380+0000","last_fullsized":"2024-04-13T03:31:43.562380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:41.895243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562374+0000","last_change":"2024-04-13T03:31:37.580598+0000","last_active":"2024-04-13T03:31:43.562374+0000","last_peered":"2024-04-13T03:31:43.562374+0000","last_clean":"2024-04-13T03:31:43.562374+0000","last_became_active":"2024-04-13T03:31:37.580276+0000","last_became_peered":"2024-04-13T03:31:37.580276+0000","last_unstale":"2024-04-13T03:31:43.562374+0000","last_undegraded":"2024-04-13T03:31:43.562374+0000","last_fullsized":"2024-04-13T03:31:43.562374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:56:02.302300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.562697+0000","last_change":"2024-04-13T03:31:37.596435+0000","last_active":"2024-04-13T03:31:43.562697+0000","last_peered":"2024-04-13T03:31:43.562697+0000","last_clean":"2024-04-13T03:31:43.562697+0000","last_became_active":"2024-04-13T03:31:37.596095+0000","last_became_peered":"2024-04-13T03:31:37.596095+0000","last_unstale":"2024-04-13T03:31:43.562697+0000","last_undegraded":"2024-04-13T03:31:43.562697+0000","last_fullsized":"2024-04-13T03:31:43.562697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:04.425037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846887+0000","last_change":"2024-04-13T03:31:37.578003+0000","last_active":"2024-04-13T03:31:42.846887+0000","last_peered":"2024-04-13T03:31:42.846887+0000","last_clean":"2024-04-13T03:31:42.846887+0000","last_became_active":"2024-04-13T03:31:37.577637+0000","last_became_peered":"2024-04-13T03:31:37.577637+0000","last_unstale":"2024-04-13T03:31:42.846887+0000","last_undegraded":"2024-04-13T03:31:42.846887+0000","last_fullsized":"2024-04-13T03:31:42.846887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:30:37.138611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563816+0000","last_change":"2024-04-13T03:31:37.585404+0000","last_active":"2024-04-13T03:31:43.563816+0000","last_peered":"2024-04-13T03:31:43.563816+0000","last_clean":"2024-04-13T03:31:43.563816+0000","last_became_active":"2024-04-13T03:31:37.585060+0000","last_became_peered":"2024-04-13T03:31:37.585060+0000","last_unstale":"2024-04-13T03:31:43.563816+0000","last_undegraded":"2024-04-13T03:31:43.563816+0000","last_fullsized":"2024-04-13T03:31:43.563816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:37:52.165706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563001+0000","last_change":"2024-04-13T03:31:37.576265+0000","last_active":"2024-04-13T03:31:43.563001+0000","last_peered":"2024-04-13T03:31:43.563001+0000","last_clean":"2024-04-13T03:31:43.563001+0000","last_became_active":"2024-04-13T03:31:37.575906+0000","last_became_peered":"2024-04-13T03:31:37.575906+0000","last_unstale":"2024-04-13T03:31:43.563001+0000","last_undegraded":"2024-04-13T03:31:43.563001+0000","last_fullsized":"2024-04-13T03:31:43.563001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:31:34.505830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.250200+0000","last_change":"2024-04-13T03:31:37.582865+0000","last_active":"2024-04-13T03:31:43.250200+0000","last_peered":"2024-04-13T03:31:43.250200+0000","last_clean":"2024-04-13T03:31:43.250200+0000","last_became_active":"2024-04-13T03:31:37.581055+0000","last_became_peered":"2024-04-13T03:31:37.581055+0000","last_unstale":"2024-04-13T03:31:43.250200+0000","last_undegraded":"2024-04-13T03:31:43.250200+0000","last_fullsized":"2024-04-13T03:31:43.250200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:01:07.815483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.834009+0000","last_change":"2024-04-13T03:31:37.587677+0000","last_active":"2024-04-13T03:31:42.834009+0000","last_peered":"2024-04-13T03:31:42.834009+0000","last_clean":"2024-04-13T03:31:42.834009+0000","last_became_active":"2024-04-13T03:31:37.586778+0000","last_became_peered":"2024-04-13T03:31:37.586778+0000","last_unstale":"2024-04-13T03:31:42.834009+0000","last_undegraded":"2024-04-13T03:31:42.834009+0000","last_fullsized":"2024-04-13T03:31:42.834009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:52:13.968559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.248625+0000","last_change":"2024-04-13T03:31:37.577694+0000","last_active":"2024-04-13T03:31:43.248625+0000","last_peered":"2024-04-13T03:31:43.248625+0000","last_clean":"2024-04-13T03:31:43.248625+0000","last_became_active":"2024-04-13T03:31:37.577287+0000","last_became_peered":"2024-04-13T03:31:37.577287+0000","last_unstale":"2024-04-13T03:31:43.248625+0000","last_undegraded":"2024-04-13T03:31:43.248625+0000","last_fullsized":"2024-04-13T03:31:43.248625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:16:26.503330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.422645+0000","last_change":"2024-04-13T03:31:37.591160+0000","last_active":"2024-04-13T03:31:43.422645+0000","last_peered":"2024-04-13T03:31:43.422645+0000","last_clean":"2024-04-13T03:31:43.422645+0000","last_became_active":"2024-04-13T03:31:37.590666+0000","last_became_peered":"2024-04-13T03:31:37.590666+0000","last_unstale":"2024-04-13T03:31:43.422645+0000","last_undegraded":"2024-04-13T03:31:43.422645+0000","last_fullsized":"2024-04-13T03:31:43.422645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:43:25.976108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:42.846756+0000","last_change":"2024-04-13T03:31:37.594433+0000","last_active":"2024-04-13T03:31:42.846756+0000","last_peered":"2024-04-13T03:31:42.846756+0000","last_clean":"2024-04-13T03:31:42.846756+0000","last_became_active":"2024-04-13T03:31:37.590942+0000","last_became_peered":"2024-04-13T03:31:37.590942+0000","last_unstale":"2024-04-13T03:31:42.846756+0000","last_undegraded":"2024-04-13T03:31:42.846756+0000","last_fullsized":"2024-04-13T03:31:42.846756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:42:47.534702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-13T03:31:43.563560+0000","last_change":"2024-04-13T03:31:37.583342+0000","last_active":"2024-04-13T03:31:43.563560+0000","last_peered":"2024-04-13T03:31:43.563560+0000","last_clean":"2024-04-13T03:31:43.563560+0000","last_became_active":"2024-04-13T03:31:37.582983+0000","last_became_peered":"2024-04-13T03:31:37.582983+0000","last_unstale":"2024-04-13T03:31:43.563560+0000","last_undegraded":"2024-04-13T03:31:43.563560+0000","last_fullsized":"2024-04-13T03:31:43.563560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:24:26.616843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284330,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":47335,"num_objects":77,"num_object_clones":0,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":22,"num_read_kb":57,"num_write":177,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":113697,"data_compressed":3654,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350996,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":489158,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612633,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1118208,"data_stored":1118271,"data_compressed":52693,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":484688,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":484608,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1077484,"data_compressed":50866,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":484592,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":508763,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":486048,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":489684,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":484300,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920826,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":484536,"data_compressed":14709,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":1081545,"data_compressed":50866,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":43658,"data_compressed":1827,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":44033,"data_compressed":1827,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T03:32:10.977 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-13T03:32:10.977 INFO:tasks.cephadm:Setting up client nodes... 2024-04-13T03:32:10.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-13T03:32:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:11 smithi016 ceph-mon[31772]: pgmap v335: 129 pgs: 129 active+clean; 623 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.5 KiB/s wr, 12 op/s 2024-04-13T03:32:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:11 smithi016 ceph-mon[31772]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:32:11.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:11 smithi188 ceph-mon[36398]: pgmap v335: 129 pgs: 129 active+clean; 623 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.5 KiB/s wr, 12 op/s 2024-04-13T03:32:11.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:11 smithi188 ceph-mon[36398]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:32:11.499 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:11.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:11 smithi119 ceph-mon[35835]: pgmap v335: 129 pgs: 129 active+clean; 623 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7.5 KiB/s wr, 12 op/s 2024-04-13T03:32:11.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:11 smithi119 ceph-mon[35835]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:32:12.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:12 smithi016 ceph-mon[31772]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:32:12.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:12 smithi188 ceph-mon[36398]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:32:12.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:12 smithi119 ceph-mon[35835]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:32:12.821 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:32:12 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: 2024-04-13T03:32:12.513+0000 7f7758253700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-13T03:32:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:13 smithi119 ceph-mon[35835]: pgmap v336: 129 pgs: 129 active+clean; 624 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-04-13T03:32:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:13 smithi016 ceph-mon[31772]: pgmap v336: 129 pgs: 129 active+clean; 624 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-04-13T03:32:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:13 smithi188 ceph-mon[36398]: pgmap v336: 129 pgs: 129 active+clean; 624 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-04-13T03:32:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.978+0000 7f0185800700 1 -- 172.21.15.16:0/3630432127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0180105f50 msgr2=0x7f01801063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.978+0000 7f0185800700 1 --2- 172.21.15.16:0/3630432127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0180105f50 0x7f01801063b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f016c009990 tx=0x7f016c009ca0 comp rx=0 tx=0).stop 2024-04-13T03:32:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.978+0000 7f0185800700 1 -- 172.21.15.16:0/3630432127 shutdown_connections 2024-04-13T03:32:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.978+0000 7f0185800700 1 --2- 172.21.15.16:0/3630432127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01801068f0 0x7f018010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.978+0000 7f0185800700 1 --2- 172.21.15.16:0/3630432127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0180104d50 0x7f0180105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.978+0000 7f0185800700 1 --2- 172.21.15.16:0/3630432127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0180105f50 0x7f01801063b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.978+0000 7f0185800700 1 -- 172.21.15.16:0/3630432127 >> 172.21.15.16:0/3630432127 conn(0x7f0180100310 msgr2=0x7f0180102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.978+0000 7f0185800700 1 -- 172.21.15.16:0/3630432127 shutdown_connections 2024-04-13T03:32:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.978+0000 7f0185800700 1 -- 172.21.15.16:0/3630432127 wait complete. 2024-04-13T03:32:13.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.980+0000 7f0185800700 1 Processor -- start 2024-04-13T03:32:13.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.980+0000 7f0185800700 1 -- start start 2024-04-13T03:32:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.980+0000 7f0185800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0180104d50 0x7f018019ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.980+0000 7f0185800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0180105f50 0x7f018019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.980+0000 7f0185800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01801068f0 0x7f01801a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.980+0000 7f0185800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0180110930 con 0x7f01801068f0 2024-04-13T03:32:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.980+0000 7f0185800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f01801107b0 con 0x7f0180105f50 2024-04-13T03:32:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.980+0000 7f0185800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0180110ab0 con 0x7f0180104d50 2024-04-13T03:32:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01801068f0 0x7f01801a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:13.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017e000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0180105f50 0x7f018019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0180104d50 0x7f018019ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017e000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0180105f50 0x7f018019b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:47208/0 (socket says 172.21.15.16:47208) 2024-04-13T03:32:13.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017e000700 1 -- 172.21.15.16:0/3736186757 learned_addr learned my addr 172.21.15.16:0/3736186757 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017ea00700 1 -- 172.21.15.16:0/3736186757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0180105f50 msgr2=0x7f018019b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017ea00700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0180105f50 0x7f018019b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017ea00700 1 -- 172.21.15.16:0/3736186757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01801068f0 msgr2=0x7f01801a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017ea00700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01801068f0 0x7f01801a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.981+0000 7f017ea00700 1 -- 172.21.15.16:0/3736186757 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01801a2ab0 con 0x7f0180104d50 2024-04-13T03:32:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.982+0000 7f017ea00700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0180104d50 0x7f018019ae50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f017000e9f0 tx=0x7f017000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.982+0000 7f0177400700 1 -- 172.21.15.16:0/3736186757 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0170004230 con 0x7f0180104d50 2024-04-13T03:32:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.982+0000 7f0177400700 1 -- 172.21.15.16:0/3736186757 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01700043d0 con 0x7f0180104d50 2024-04-13T03:32:13.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.982+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01801a2da0 con 0x7f0180104d50 2024-04-13T03:32:13.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.982+0000 7f0177400700 1 -- 172.21.15.16:0/3736186757 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0170010430 con 0x7f0180104d50 2024-04-13T03:32:13.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.982+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01801a31f0 con 0x7f0180104d50 2024-04-13T03:32:13.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.983+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f018002dfa0 con 0x7f0180104d50 2024-04-13T03:32:13.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.985+0000 7f0177400700 1 -- 172.21.15.16:0/3736186757 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f017000c7c0 con 0x7f0180104d50 2024-04-13T03:32:13.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.987+0000 7f0177400700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01640a3e00 0x7f01640a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:13.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.987+0000 7f0177400700 1 -- 172.21.15.16:0/3736186757 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f01700d0a70 con 0x7f0180104d50 2024-04-13T03:32:13.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.987+0000 7f017e000700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01640a3e00 0x7f01640a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:13.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.988+0000 7f017e000700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01640a3e00 0x7f01640a62c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f016c00b2e0 tx=0x7f016c00b8f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:13.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:13.992+0000 7f0177400700 1 -- 172.21.15.16:0/3736186757 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f017009d050 con 0x7f0180104d50 2024-04-13T03:32:14.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.319+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f018002e1c0 con 0x7f0180104d50 2024-04-13T03:32:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.331+0000 7f0177400700 1 -- 172.21.15.16:0/3736186757 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f017009c950 con 0x7f0180104d50 2024-04-13T03:32:14.331 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-04-13T03:32:14.331 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQC+/BlmorNAExAAVnSTHLzmputEy9eqap1ewQ== 2024-04-13T03:32:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01640a3e00 msgr2=0x7f01640a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01640a3e00 0x7f01640a62c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f016c00b2e0 tx=0x7f016c00b8f0 comp rx=0 tx=0).stop 2024-04-13T03:32:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0180104d50 msgr2=0x7f018019ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0180104d50 0x7f018019ae50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f017000e9f0 tx=0x7f017000ed00 comp rx=0 tx=0).stop 2024-04-13T03:32:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 shutdown_connections 2024-04-13T03:32:14.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01640a3e00 0x7f01640a62c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01801068f0 0x7f01801a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0180104d50 0x7f018019ae50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 --2- 172.21.15.16:0/3736186757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0180105f50 0x7f018019b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 >> 172.21.15.16:0/3736186757 conn(0x7f0180100310 msgr2=0x7f0180109180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 shutdown_connections 2024-04-13T03:32:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:14.333+0000 7f0185800700 1 -- 172.21.15.16:0/3736186757 wait complete. 2024-04-13T03:32:15.192 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:32:15.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-13T03:32:15.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-13T03:32:15.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:15 smithi016 ceph-mon[31772]: pgmap v337: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.2 KiB/s wr, 20 op/s 2024-04-13T03:32:15.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:15 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3736186757' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-13T03:32:15.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:15 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-13T03:32:15.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:15 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-13T03:32:15.357 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-13T03:32:15.357 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-13T03:32:15.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph mgr dump --format=json 2024-04-13T03:32:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:15 smithi188 ceph-mon[36398]: pgmap v337: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.2 KiB/s wr, 20 op/s 2024-04-13T03:32:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:15 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3736186757' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-13T03:32:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:15 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-13T03:32:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:15 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-13T03:32:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:15 smithi119 ceph-mon[35835]: pgmap v337: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.2 KiB/s wr, 20 op/s 2024-04-13T03:32:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:15 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3736186757' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-13T03:32:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:15 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-13T03:32:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:15 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-13T03:32:15.881 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:17.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:17 smithi016 ceph-mon[31772]: pgmap v338: 129 pgs: 129 active+clean; 629 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.6 KiB/s wr, 22 op/s 2024-04-13T03:32:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:17 smithi188 ceph-mon[36398]: pgmap v338: 129 pgs: 129 active+clean; 629 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.6 KiB/s wr, 22 op/s 2024-04-13T03:32:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:17 smithi119 ceph-mon[35835]: pgmap v338: 129 pgs: 129 active+clean; 629 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.6 KiB/s wr, 22 op/s 2024-04-13T03:32:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.274+0000 7f2baee00700 1 -- 172.21.15.16:0/2364411859 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8113860 msgr2=0x7f2ba8115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.274+0000 7f2baee00700 1 --2- 172.21.15.16:0/2364411859 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8113860 0x7f2ba8115c50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000b0a0 tx=0x7f2ba000b3b0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.275+0000 7f2baee00700 1 -- 172.21.15.16:0/2364411859 shutdown_connections 2024-04-13T03:32:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.275+0000 7f2baee00700 1 --2- 172.21.15.16:0/2364411859 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8113860 0x7f2ba8115c50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.275+0000 7f2baee00700 1 --2- 172.21.15.16:0/2364411859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba8076fe0 0x7f2ba80773c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.275+0000 7f2baee00700 1 --2- 172.21.15.16:0/2364411859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8077900 0x7f2ba8113320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.275+0000 7f2baee00700 1 -- 172.21.15.16:0/2364411859 >> 172.21.15.16:0/2364411859 conn(0x7f2ba81006f0 msgr2=0x7f2ba8102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:18.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.275+0000 7f2baee00700 1 -- 172.21.15.16:0/2364411859 shutdown_connections 2024-04-13T03:32:18.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.275+0000 7f2baee00700 1 -- 172.21.15.16:0/2364411859 wait complete. 2024-04-13T03:32:18.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.276+0000 7f2baee00700 1 Processor -- start 2024-04-13T03:32:18.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.277+0000 7f2baee00700 1 -- start start 2024-04-13T03:32:18.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.277+0000 7f2baee00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8076fe0 0x7f2ba81a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:18.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.277+0000 7f2baee00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba8077900 0x7f2ba81a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:18.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.277+0000 7f2baee00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8113860 0x7f2ba819f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:18.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.277+0000 7f2baee00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba8119330 con 0x7f2ba8113860 2024-04-13T03:32:18.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.277+0000 7f2baee00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81191b0 con 0x7f2ba8076fe0 2024-04-13T03:32:18.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.277+0000 7f2baee00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba81194b0 con 0x7f2ba8077900 2024-04-13T03:32:18.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bada00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8076fe0 0x7f2ba81a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:18.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bada00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8076fe0 0x7f2ba81a53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:47228/0 (socket says 172.21.15.16:47228) 2024-04-13T03:32:18.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bada00700 1 -- 172.21.15.16:0/2169477818 learned_addr learned my addr 172.21.15.16:0/2169477818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:18.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bae400700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8113860 0x7f2ba819f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:18.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bad000700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba8077900 0x7f2ba81a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:18.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bada00700 1 -- 172.21.15.16:0/2169477818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba8077900 msgr2=0x7f2ba81a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:18.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bada00700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba8077900 0x7f2ba81a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bada00700 1 -- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8113860 msgr2=0x7f2ba819f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:18.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bada00700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8113860 0x7f2ba819f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.278+0000 7f2bada00700 1 -- 172.21.15.16:0/2169477818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba819fd10 con 0x7f2ba8076fe0 2024-04-13T03:32:18.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.279+0000 7f2bada00700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8076fe0 0x7f2ba81a53f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00e9a0 tx=0x7f2b9c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.279+0000 7f2ba6000700 1 -- 172.21.15.16:0/2169477818 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c00ca10 con 0x7f2ba8076fe0 2024-04-13T03:32:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.279+0000 7f2ba6000700 1 -- 172.21.15.16:0/2169477818 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b9c00cbb0 con 0x7f2ba8076fe0 2024-04-13T03:32:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.279+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba81a0000 con 0x7f2ba8076fe0 2024-04-13T03:32:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.279+0000 7f2ba6000700 1 -- 172.21.15.16:0/2169477818 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c018680 con 0x7f2ba8076fe0 2024-04-13T03:32:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.279+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba81ac2c0 con 0x7f2ba8076fe0 2024-04-13T03:32:18.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.280+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba80697a0 con 0x7f2ba8076fe0 2024-04-13T03:32:18.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.282+0000 7f2ba6000700 1 -- 172.21.15.16:0/2169477818 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2b9c010430 con 0x7f2ba8076fe0 2024-04-13T03:32:18.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.283+0000 7f2ba6000700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b900a3d50 0x7f2b900a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:18.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.284+0000 7f2ba6000700 1 -- 172.21.15.16:0/2169477818 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2b9c0d09c0 con 0x7f2ba8076fe0 2024-04-13T03:32:18.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.284+0000 7f2bad000700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b900a3d50 0x7f2b900a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:18.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.285+0000 7f2bad000700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b900a3d50 0x7f2b900a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8104150 tx=0x7f2b94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:18.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.286+0000 7f2ba6000700 1 -- 172.21.15.16:0/2169477818 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b9c09cfa0 con 0x7f2ba8076fe0 2024-04-13T03:32:18.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.582+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f2ba802dfa0 con 0x7f2ba8076fe0 2024-04-13T03:32:18.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.588+0000 7f2ba6000700 1 -- 172.21.15.16:0/2169477818 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264998 (secure 0 0 0) 0x7f2b9c09c8a0 con 0x7f2ba8076fe0 2024-04-13T03:32:18.594 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:32:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b900a3d50 msgr2=0x7f2b900a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b900a3d50 0x7f2b900a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8104150 tx=0x7f2b94009260 comp rx=0 tx=0).stop 2024-04-13T03:32:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8076fe0 msgr2=0x7f2ba81a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8076fe0 0x7f2ba81a53f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00e9a0 tx=0x7f2b9c00ecb0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 shutdown_connections 2024-04-13T03:32:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b900a3d50 0x7f2b900a6210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba8113860 0x7f2ba819f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba8076fe0 0x7f2ba81a53f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 --2- 172.21.15.16:0/2169477818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba8077900 0x7f2ba81a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.595+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 >> 172.21.15.16:0/2169477818 conn(0x7f2ba81006f0 msgr2=0x7f2ba8102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.596+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 shutdown_connections 2024-04-13T03:32:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:18.596+0000 7f2baee00700 1 -- 172.21.15.16:0/2169477818 wait complete. 2024-04-13T03:32:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:19 smithi188 ceph-mon[36398]: pgmap v339: 129 pgs: 129 active+clean; 629 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-04-13T03:32:19.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:19 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2169477818' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-13T03:32:19.397 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:19 smithi016 ceph-mon[31772]: pgmap v339: 129 pgs: 129 active+clean; 629 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-04-13T03:32:19.397 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:19 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2169477818' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-13T03:32:19.403 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":1721562410},{"type":"v1","addr":"172.21.15.16:6801","nonce":1721562410}]},"active_addr":"172.21.15.16:6801/1721562410","active_change":"2024-04-13T03:23:39.715209+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2348781121}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2756805961}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2242726589}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1183476525}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3831471228}]}]} 2024-04-13T03:32:19.409 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-13T03:32:19.409 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-13T03:32:19.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd dump --format=json 2024-04-13T03:32:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:19 smithi119 ceph-mon[35835]: pgmap v339: 129 pgs: 129 active+clean; 629 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-04-13T03:32:19.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:19 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2169477818' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-13T03:32:19.909 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:21.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:21 smithi016 ceph-mon[31772]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 21 op/s 2024-04-13T03:32:21.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:21 smithi188 ceph-mon[36398]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 21 op/s 2024-04-13T03:32:21.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:21 smithi119 ceph-mon[35835]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 21 op/s 2024-04-13T03:32:22.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.358+0000 7f370de00700 1 -- 172.21.15.16:0/1606456386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370806a7c0 msgr2=0x7f370806ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:22.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.358+0000 7f370de00700 1 --2- 172.21.15.16:0/1606456386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370806a7c0 0x7f370806ac20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009990 tx=0x7f36f4009ca0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.359+0000 7f370de00700 1 -- 172.21.15.16:0/1606456386 shutdown_connections 2024-04-13T03:32:22.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.359+0000 7f370de00700 1 --2- 172.21.15.16:0/1606456386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f370806b160 0x7f370810f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.359+0000 7f370de00700 1 --2- 172.21.15.16:0/1606456386 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3708108aa0 0x7f3708108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.359+0000 7f370de00700 1 --2- 172.21.15.16:0/1606456386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370806a7c0 0x7f370806ac20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.359+0000 7f370de00700 1 -- 172.21.15.16:0/1606456386 >> 172.21.15.16:0/1606456386 conn(0x7f3708078960 msgr2=0x7f3708078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.359+0000 7f370de00700 1 -- 172.21.15.16:0/1606456386 shutdown_connections 2024-04-13T03:32:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.359+0000 7f370de00700 1 -- 172.21.15.16:0/1606456386 wait complete. 2024-04-13T03:32:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.360+0000 7f370de00700 1 Processor -- start 2024-04-13T03:32:22.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.360+0000 7f370de00700 1 -- start start 2024-04-13T03:32:22.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.360+0000 7f370de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f370806a7c0 0x7f370819b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:22.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.360+0000 7f370de00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370806b160 0x7f370819ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:22.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.360+0000 7f370de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3708108aa0 0x7f370819fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:22.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.360+0000 7f370de00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3708112de0 con 0x7f370806b160 2024-04-13T03:32:22.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.360+0000 7f370de00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3708112c60 con 0x7f370806a7c0 2024-04-13T03:32:22.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.360+0000 7f370de00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3708112f60 con 0x7f3708108aa0 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3706000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370806b160 0x7f370819ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3706a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f370806a7c0 0x7f370819b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3706000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370806b160 0x7f370819ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56760/0 (socket says 172.21.15.16:56760) 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3706000700 1 -- 172.21.15.16:0/3869161736 learned_addr learned my addr 172.21.15.16:0/3869161736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3707400700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3708108aa0 0x7f370819fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3706a00700 1 -- 172.21.15.16:0/3869161736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3708108aa0 msgr2=0x7f370819fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3706a00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3708108aa0 0x7f370819fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3706a00700 1 -- 172.21.15.16:0/3869161736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370806b160 msgr2=0x7f370819ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3706a00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370806b160 0x7f370819ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.361+0000 7f3706a00700 1 -- 172.21.15.16:0/3869161736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37081a0520 con 0x7f370806a7c0 2024-04-13T03:32:22.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.362+0000 7f3706a00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f370806a7c0 0x7f370819b510 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f36f800e9f0 tx=0x7f36f800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:22.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.362+0000 7f370ca00700 1 -- 172.21.15.16:0/3869161736 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8004220 con 0x7f370806a7c0 2024-04-13T03:32:22.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.362+0000 7f370ca00700 1 -- 172.21.15.16:0/3869161736 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36f8004d10 con 0x7f370806a7c0 2024-04-13T03:32:22.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.362+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37081a0810 con 0x7f370806a7c0 2024-04-13T03:32:22.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.362+0000 7f370ca00700 1 -- 172.21.15.16:0/3869161736 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8005020 con 0x7f370806a7c0 2024-04-13T03:32:22.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.362+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37081a8090 con 0x7f370806a7c0 2024-04-13T03:32:22.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.363+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37080697a0 con 0x7f370806a7c0 2024-04-13T03:32:22.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.365+0000 7f370ca00700 1 -- 172.21.15.16:0/3869161736 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f36f8004750 con 0x7f370806a7c0 2024-04-13T03:32:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.367+0000 7f370ca00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f36ec0a3cd0 0x7f36ec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.367+0000 7f370ca00700 1 -- 172.21.15.16:0/3869161736 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f36f80d0660 con 0x7f370806a7c0 2024-04-13T03:32:22.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.367+0000 7f3706000700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f36ec0a3cd0 0x7f36ec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:22.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.368+0000 7f3706000700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f36ec0a3cd0 0x7f36ec0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3708078f60 tx=0x7f36f400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:22.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.371+0000 7f370ca00700 1 -- 172.21.15.16:0/3869161736 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36f809cc40 con 0x7f370806a7c0 2024-04-13T03:32:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.605+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f370819c8a0 con 0x7f370806a7c0 2024-04-13T03:32:22.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.607+0000 7f370ca00700 1 -- 172.21.15.16:0/3869161736 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7f370819c8a0 con 0x7f370806a7c0 2024-04-13T03:32:22.607 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:32:22.608 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:32:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.611+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f36ec0a3cd0 msgr2=0x7f36ec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:22.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.611+0000 7f370de00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f36ec0a3cd0 0x7f36ec0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3708078f60 tx=0x7f36f400b260 comp rx=0 tx=0).stop 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.611+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f370806a7c0 msgr2=0x7f370819b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.611+0000 7f370de00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f370806a7c0 0x7f370819b510 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f36f800e9f0 tx=0x7f36f800ed00 comp rx=0 tx=0).stop 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.612+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 shutdown_connections 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.612+0000 7f370de00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f36ec0a3cd0 0x7f36ec0a6190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.612+0000 7f370de00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3708108aa0 0x7f370819fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.612+0000 7f370de00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f370806a7c0 0x7f370819b510 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.612+0000 7f370de00700 1 --2- 172.21.15.16:0/3869161736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370806b160 0x7f370819ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.612+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 >> 172.21.15.16:0/3869161736 conn(0x7f3708078960 msgr2=0x7f370810df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.612+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 shutdown_connections 2024-04-13T03:32:22.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:22.612+0000 7f370de00700 1 -- 172.21.15.16:0/3869161736 wait complete. 2024-04-13T03:32:23.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:23 smithi188 ceph-mon[36398]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 2.6 KiB/s wr, 16 op/s 2024-04-13T03:32:23.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:23 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3869161736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:23.522 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:23 smithi016 ceph-mon[31772]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 2.6 KiB/s wr, 16 op/s 2024-04-13T03:32:23.523 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:23 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3869161736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:23.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:23 smithi119 ceph-mon[35835]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 2.6 KiB/s wr, 16 op/s 2024-04-13T03:32:23.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:23 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3869161736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:23.579 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-13T03:32:23.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd dump --format=json 2024-04-13T03:32:24.082 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:25.141 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:25 smithi016 ceph-mon[31772]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-04-13T03:32:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:25 smithi119 ceph-mon[35835]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-04-13T03:32:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:25 smithi188 ceph-mon[36398]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-04-13T03:32:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.295+0000 7f0165c00700 1 -- 172.21.15.16:0/3458523413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0160075600 msgr2=0x7f016010efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.295+0000 7f0165c00700 1 --2- 172.21.15.16:0/3458523413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0160075600 0x7f016010efe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f01480098f0 tx=0x7f0148009c00 comp rx=0 tx=0).stop 2024-04-13T03:32:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.296+0000 7f0165c00700 1 -- 172.21.15.16:0/3458523413 shutdown_connections 2024-04-13T03:32:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.296+0000 7f0165c00700 1 --2- 172.21.15.16:0/3458523413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f016010f520 0x7f0160111910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.296+0000 7f0165c00700 1 --2- 172.21.15.16:0/3458523413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0160074ce0 0x7f01600750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.296+0000 7f0165c00700 1 --2- 172.21.15.16:0/3458523413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0160075600 0x7f016010efe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.296+0000 7f0165c00700 1 -- 172.21.15.16:0/3458523413 >> 172.21.15.16:0/3458523413 conn(0x7f0160079c70 msgr2=0x7f016007a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:26.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.296+0000 7f0165c00700 1 -- 172.21.15.16:0/3458523413 shutdown_connections 2024-04-13T03:32:26.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.296+0000 7f0165c00700 1 -- 172.21.15.16:0/3458523413 wait complete. 2024-04-13T03:32:26.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.297+0000 7f0165c00700 1 Processor -- start 2024-04-13T03:32:26.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.297+0000 7f0165c00700 1 -- start start 2024-04-13T03:32:26.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f0165c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0160074ce0 0x7f016019b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:26.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f0165c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0160075600 0x7f016019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:26.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f0165c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f016010f520 0x7f016019fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:26.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f0165c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0160114fe0 con 0x7f016010f520 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f0165c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0160114e60 con 0x7f0160074ce0 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f0165c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0160115160 con 0x7f0160075600 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f015f400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f016010f520 0x7f016019fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f015ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0160074ce0 0x7f016019b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f015ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0160074ce0 0x7f016019b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:55352/0 (socket says 172.21.15.16:55352) 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f015ea00700 1 -- 172.21.15.16:0/3747446980 learned_addr learned my addr 172.21.15.16:0/3747446980 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f0157e00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0160075600 0x7f016019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f015ea00700 1 -- 172.21.15.16:0/3747446980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0160075600 msgr2=0x7f016019ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f015ea00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0160075600 0x7f016019ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f015ea00700 1 -- 172.21.15.16:0/3747446980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f016010f520 msgr2=0x7f016019fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f015ea00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f016010f520 0x7f016019fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.298+0000 7f015ea00700 1 -- 172.21.15.16:0/3747446980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01601a0520 con 0x7f0160074ce0 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.299+0000 7f015ea00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0160074ce0 0x7f016019b510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f015000e9c0 tx=0x7f015000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:26.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.299+0000 7f0157400700 1 -- 172.21.15.16:0/3747446980 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150004a30 con 0x7f0160074ce0 2024-04-13T03:32:26.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.299+0000 7f0157400700 1 -- 172.21.15.16:0/3747446980 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015000cca0 con 0x7f0160074ce0 2024-04-13T03:32:26.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.299+0000 7f0157400700 1 -- 172.21.15.16:0/3747446980 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150005680 con 0x7f0160074ce0 2024-04-13T03:32:26.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.299+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01601a0810 con 0x7f0160074ce0 2024-04-13T03:32:26.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.299+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01601a8090 con 0x7f0160074ce0 2024-04-13T03:32:26.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.300+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01600697a0 con 0x7f0160074ce0 2024-04-13T03:32:26.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.302+0000 7f0157400700 1 -- 172.21.15.16:0/3747446980 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f015000c7c0 con 0x7f0160074ce0 2024-04-13T03:32:26.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.304+0000 7f0157400700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01440a3cd0 0x7f01440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:26.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.304+0000 7f0157400700 1 -- 172.21.15.16:0/3747446980 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f01500d0a60 con 0x7f0160074ce0 2024-04-13T03:32:26.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.304+0000 7f0157e00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01440a3cd0 0x7f01440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:26.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.305+0000 7f0157e00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01440a3cd0 0x7f01440a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0148009810 tx=0x7f0148019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:26.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.306+0000 7f0157400700 1 -- 172.21.15.16:0/3747446980 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f015009d040 con 0x7f0160074ce0 2024-04-13T03:32:26.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.534+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f016019c870 con 0x7f0160074ce0 2024-04-13T03:32:26.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.535+0000 7f0157400700 1 -- 172.21.15.16:0/3747446980 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24160 (secure 0 0 0) 0x7f015009c940 con 0x7f0160074ce0 2024-04-13T03:32:26.538 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:32:26.538 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:31:42.303791+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:32:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01440a3cd0 msgr2=0x7f01440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:26.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01440a3cd0 0x7f01440a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0148009810 tx=0x7f0148019040 comp rx=0 tx=0).stop 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0160074ce0 msgr2=0x7f016019b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0160074ce0 0x7f016019b510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f015000e9c0 tx=0x7f015000ecd0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 shutdown_connections 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f01440a3cd0 0x7f01440a6190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f016010f520 0x7f016019fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0160074ce0 0x7f016019b510 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 --2- 172.21.15.16:0/3747446980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0160075600 0x7f016019ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 >> 172.21.15.16:0/3747446980 conn(0x7f0160079c70 msgr2=0x7f01600ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 shutdown_connections 2024-04-13T03:32:26.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:26.539+0000 7f0165c00700 1 -- 172.21.15.16:0/3747446980 wait complete. 2024-04-13T03:32:27.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:27 smithi016 ceph-mon[31772]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-13T03:32:27.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:27 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3747446980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:27.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-13T03:32:27.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-13T03:32:27.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-13T03:32:27.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-13T03:32:27.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-13T03:32:27.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-13T03:32:27.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-13T03:32:27.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-13T03:32:27.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-13T03:32:27.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-13T03:32:27.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-13T03:32:27.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-13T03:32:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:27 smithi119 ceph-mon[35835]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-13T03:32:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:27 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3747446980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:27.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:27 smithi188 ceph-mon[36398]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-13T03:32:27.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:27 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3747446980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:32:28.498 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:28.500 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:28.500 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:28.500 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:28.514 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:28.518 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:28.519 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:28.566 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:29.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:29 smithi016 ceph-mon[31772]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-13T03:32:29.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:29 smithi119 ceph-mon[35835]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-13T03:32:29.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:29 smithi188 ceph-mon[36398]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-13T03:32:31.424 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:31 smithi016 ceph-mon[31772]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-13T03:32:31.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:31 smithi119 ceph-mon[35835]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-13T03:32:31.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:31 smithi188 ceph-mon[36398]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-13T03:32:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:32.071 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:32.071 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:33.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:33 smithi016 ceph-mon[31772]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-13T03:32:33.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:33 smithi119 ceph-mon[35835]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-13T03:32:33.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:33 smithi188 ceph-mon[36398]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-13T03:32:35.528 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:35 smithi016 ceph-mon[31772]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-13T03:32:35.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:35 smithi119 ceph-mon[35835]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-13T03:32:35.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:35 smithi188 ceph-mon[36398]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-13T03:32:37.489 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:37 smithi016 ceph-mon[31772]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-13T03:32:37.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:37 smithi119 ceph-mon[35835]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-13T03:32:37.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:37 smithi188 ceph-mon[36398]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-13T03:32:38.469 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:38 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:32:38.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:38 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:32:38.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:38 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:32:39.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:39 smithi016 ceph-mon[31772]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-13T03:32:39.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:39 smithi119 ceph-mon[35835]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-13T03:32:39.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:39 smithi188 ceph-mon[36398]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-13T03:32:39.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.665+0000 7f8cba600700 1 -- 172.21.15.16:0/1191169768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cb4108aa0 msgr2=0x7f8cb4108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:39.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.665+0000 7f8cba600700 1 --2- 172.21.15.16:0/1191169768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cb4108aa0 0x7f8cb4108e80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009990 tx=0x7f8ca4009ca0 comp rx=0 tx=0).stop 2024-04-13T03:32:39.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.665+0000 7f8cba600700 1 -- 172.21.15.16:0/1191169768 shutdown_connections 2024-04-13T03:32:39.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.665+0000 7f8cba600700 1 --2- 172.21.15.16:0/1191169768 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb406b160 0x7f8cb410f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:39.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.665+0000 7f8cba600700 1 --2- 172.21.15.16:0/1191169768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cb4108aa0 0x7f8cb4108e80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:39.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.665+0000 7f8cba600700 1 --2- 172.21.15.16:0/1191169768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb406a7c0 0x7f8cb406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:39.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.665+0000 7f8cba600700 1 -- 172.21.15.16:0/1191169768 >> 172.21.15.16:0/1191169768 conn(0x7f8cb4078960 msgr2=0x7f8cb4078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:39.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.665+0000 7f8cba600700 1 -- 172.21.15.16:0/1191169768 shutdown_connections 2024-04-13T03:32:39.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.665+0000 7f8cba600700 1 -- 172.21.15.16:0/1191169768 wait complete. 2024-04-13T03:32:39.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.666+0000 7f8cba600700 1 Processor -- start 2024-04-13T03:32:39.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.666+0000 7f8cba600700 1 -- start start 2024-04-13T03:32:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cba600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb406a7c0 0x7f8cb4199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cba600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb406b160 0x7f8cb41997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cba600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cb4108aa0 0x7f8cb419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cba600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4112de0 con 0x7f8cb406b160 2024-04-13T03:32:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cba600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4112c60 con 0x7f8cb4108aa0 2024-04-13T03:32:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cba600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4112f60 con 0x7f8cb406a7c0 2024-04-13T03:32:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cb3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb406a7c0 0x7f8cb4199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cb3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb406a7c0 0x7f8cb4199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:39558/0 (socket says 172.21.15.16:39558) 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cb3400700 1 -- 172.21.15.16:0/3255056495 learned_addr learned my addr 172.21.15.16:0/3255056495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cb2a00700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb406b160 0x7f8cb41997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.667+0000 7f8cb3e00700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cb4108aa0 0x7f8cb419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.668+0000 7f8cb2a00700 1 -- 172.21.15.16:0/3255056495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb406a7c0 msgr2=0x7f8cb4199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.668+0000 7f8cb2a00700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb406a7c0 0x7f8cb4199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.668+0000 7f8cb2a00700 1 -- 172.21.15.16:0/3255056495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cb4108aa0 msgr2=0x7f8cb419db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.668+0000 7f8cb2a00700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cb4108aa0 0x7f8cb419db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.668+0000 7f8cb2a00700 1 -- 172.21.15.16:0/3255056495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb419e280 con 0x7f8cb406b160 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.668+0000 7f8cb3400700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cb406a7c0 0x7f8cb4199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.668+0000 7f8cb2a00700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb406b160 0x7f8cb41997b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000e9f0 tx=0x7f8ca000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.669+0000 7f8cabe00700 1 -- 172.21.15.16:0/3255056495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca000c9c0 con 0x7f8cb406b160 2024-04-13T03:32:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.669+0000 7f8cabe00700 1 -- 172.21.15.16:0/3255056495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ca000cb60 con 0x7f8cb406b160 2024-04-13T03:32:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.669+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb419e570 con 0x7f8cb406b160 2024-04-13T03:32:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.669+0000 7f8cabe00700 1 -- 172.21.15.16:0/3255056495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0018680 con 0x7f8cb406b160 2024-04-13T03:32:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.669+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb41a5df0 con 0x7f8cb406b160 2024-04-13T03:32:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.670+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8cb4063240 con 0x7f8cb406b160 2024-04-13T03:32:39.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.673+0000 7f8cabe00700 1 -- 172.21.15.16:0/3255056495 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8ca0010430 con 0x7f8cb406b160 2024-04-13T03:32:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.676+0000 7f8cabe00700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8c980a3cd0 0x7f8c980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.676+0000 7f8cabe00700 1 -- 172.21.15.16:0/3255056495 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8ca00d0d80 con 0x7f8cb406b160 2024-04-13T03:32:39.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.676+0000 7f8cb3400700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8c980a3cd0 0x7f8c980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.676+0000 7f8cabe00700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] conn(0x7f8c980ae990 0x7f8c980b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.676+0000 7f8cabe00700 1 -- 172.21.15.16:0/3255056495 --> [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ca0019ec0 con 0x7f8c980ae990 2024-04-13T03:32:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.676+0000 7f8cabe00700 1 -- 172.21.15.16:0/3255056495 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f8ca00d1130 con 0x7f8cb406b160 2024-04-13T03:32:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.677+0000 7f8cb3e00700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] conn(0x7f8c980ae990 0x7f8c980b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:39.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.677+0000 7f8cb3400700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8c980a3cd0 0x7f8c980a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009750 tx=0x7f8ca400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:39.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.678+0000 7f8cb3e00700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] conn(0x7f8c980ae990 0x7f8c980b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:39.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.679+0000 7f8cabe00700 1 -- 172.21.15.16:0/3255056495 <== osd.6 v2:172.21.15.16:6818/1633490210 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ca0019ec0 con 0x7f8c980ae990 2024-04-13T03:32:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.701+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 --> [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8cb40636c0 con 0x7f8c980ae990 2024-04-13T03:32:39.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.701+0000 7f8cabe00700 1 -- 172.21.15.16:0/3255056495 <== osd.6 v2:172.21.15.16:6818/1633490210 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8cb40636c0 con 0x7f8c980ae990 2024-04-13T03:32:39.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] conn(0x7f8c980ae990 msgr2=0x7f8c980b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:39.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] conn(0x7f8c980ae990 0x7f8c980b0dd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:39.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8c980a3cd0 msgr2=0x7f8c980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:39.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8c980a3cd0 0x7f8c980a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009750 tx=0x7f8ca400b280 comp rx=0 tx=0).stop 2024-04-13T03:32:39.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb406b160 msgr2=0x7f8cb41997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:39.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 --2- 172.21.15.16:0/3255056495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cb406b160 0x7f8cb41997b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000e9f0 tx=0x7f8ca000ed00 comp rx=0 tx=0).stop 2024-04-13T03:32:39.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cb3e00700 1 -- 172.21.15.16:0/3255056495 reap_dead start 2024-04-13T03:32:39.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 shutdown_connections 2024-04-13T03:32:39.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 >> 172.21.15.16:0/3255056495 conn(0x7f8cb4078960 msgr2=0x7f8cb40ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:39.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 shutdown_connections 2024-04-13T03:32:39.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.702+0000 7f8cba600700 1 -- 172.21.15.16:0/3255056495 wait complete. 2024-04-13T03:32:40.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:40.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:40.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:40.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-13T03:32:40.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:40 smithi016 ceph-mon[31772]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-13T03:32:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-04-13T03:32:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-13T03:32:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-13T03:32:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-13T03:32:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-13T03:32:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-13T03:32:41.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-13T03:32:41.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-04-13T03:32:41.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-13T03:32:41.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-13T03:32:41.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:40 smithi119 ceph-mon[35835]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-13T03:32:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-13T03:32:41.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-13T03:32:41.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-04-13T03:32:41.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-13T03:32:41.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-13T03:32:41.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:40 smithi188 ceph-mon[36398]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-13T03:32:41.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.977+0000 7f2baae00700 1 -- 172.21.15.16:0/383497367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba4109340 msgr2=0x7f2ba4109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:41.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.977+0000 7f2baae00700 1 --2- 172.21.15.16:0/383497367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba4109340 0x7f2ba4109720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2b940099c0 tx=0x7f2b94009cd0 comp rx=0 tx=0).stop 2024-04-13T03:32:41.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.977+0000 7f2baae00700 1 -- 172.21.15.16:0/383497367 shutdown_connections 2024-04-13T03:32:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.977+0000 7f2baae00700 1 --2- 172.21.15.16:0/383497367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba410e420 0x7f2ba4115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.977+0000 7f2baae00700 1 --2- 172.21.15.16:0/383497367 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba4109340 0x7f2ba4109720 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.977+0000 7f2baae00700 1 --2- 172.21.15.16:0/383497367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba4109cf0 0x7f2ba410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.977+0000 7f2baae00700 1 -- 172.21.15.16:0/383497367 >> 172.21.15.16:0/383497367 conn(0x7f2ba4076280 msgr2=0x7f2ba40786a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.977+0000 7f2baae00700 1 -- 172.21.15.16:0/383497367 shutdown_connections 2024-04-13T03:32:41.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.977+0000 7f2baae00700 1 -- 172.21.15.16:0/383497367 wait complete. 2024-04-13T03:32:41.980 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-04-13T03:32:41.980 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-13T03:32:41.980 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-13T03:32:41.980 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-13T03:32:41.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-13T03:32:41.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-13T03:32:41.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-13T03:32:41.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-04-13T03:32:41.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-13T03:32:41.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-13T03:32:41.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:41 smithi016 ceph-mon[31772]: osdmap e91: 12 total, 12 up, 12 in 2024-04-13T03:32:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.979+0000 7f2baae00700 1 Processor -- start 2024-04-13T03:32:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.979+0000 7f2baae00700 1 -- start start 2024-04-13T03:32:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2baae00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba4109340 0x7f2ba41a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:41.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2baae00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba4109cf0 0x7f2ba41a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2baae00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba410e420 0x7f2ba419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2baae00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba4119470 con 0x7f2ba4109cf0 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2baae00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba41192f0 con 0x7f2ba410e420 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2baae00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba41195f0 con 0x7f2ba4109340 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2ba3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba4109cf0 0x7f2ba41a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2ba3400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba4109cf0 0x7f2ba41a5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55484/0 (socket says 172.21.15.16:55484) 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2ba3400700 1 -- 172.21.15.16:0/4240222347 learned_addr learned my addr 172.21.15.16:0/4240222347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2ba3e00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba4109340 0x7f2ba41a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.980+0000 7f2ba8e00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba410e420 0x7f2ba419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.981+0000 7f2ba3400700 1 -- 172.21.15.16:0/4240222347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba4109340 msgr2=0x7f2ba41a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.981+0000 7f2ba3400700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ba4109340 0x7f2ba41a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:41.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.981+0000 7f2ba3400700 1 -- 172.21.15.16:0/4240222347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba410e420 msgr2=0x7f2ba419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.981+0000 7f2ba3400700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba410e420 0x7f2ba419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.981+0000 7f2ba3400700 1 -- 172.21.15.16:0/4240222347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba41a0050 con 0x7f2ba4109cf0 2024-04-13T03:32:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.981+0000 7f2ba3400700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba4109cf0 0x7f2ba41a5be0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000e9f0 tx=0x7f2b9000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.982+0000 7f2ba0c00700 1 -- 172.21.15.16:0/4240222347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90004a30 con 0x7f2ba4109cf0 2024-04-13T03:32:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.982+0000 7f2ba0c00700 1 -- 172.21.15.16:0/4240222347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b9000cca0 con 0x7f2ba4109cf0 2024-04-13T03:32:41.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.982+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba41a0340 con 0x7f2ba4109cf0 2024-04-13T03:32:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.982+0000 7f2ba0c00700 1 -- 172.21.15.16:0/4240222347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b900056c0 con 0x7f2ba4109cf0 2024-04-13T03:32:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.982+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba41ac620 con 0x7f2ba4109cf0 2024-04-13T03:32:41.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.983+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2ba4063200 con 0x7f2ba4109cf0 2024-04-13T03:32:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.987+0000 7f2ba0c00700 1 -- 172.21.15.16:0/4240222347 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2b9000c7c0 con 0x7f2ba4109cf0 2024-04-13T03:32:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.989+0000 7f2ba0c00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b880a3cd0 0x7f2b880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.989+0000 7f2ba0c00700 1 -- 172.21.15.16:0/4240222347 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f2b900d0b30 con 0x7f2ba4109cf0 2024-04-13T03:32:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.989+0000 7f2ba3e00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b880a3cd0 0x7f2b880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.990+0000 7f2ba0c00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] conn(0x7f2b880aee30 0x7f2b880b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.990+0000 7f2ba0c00700 1 -- 172.21.15.16:0/4240222347 --> [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b90019e50 con 0x7f2b880aee30 2024-04-13T03:32:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.990+0000 7f2ba0c00700 1 -- 172.21.15.16:0/4240222347 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f2b900d52a0 con 0x7f2ba4109cf0 2024-04-13T03:32:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.990+0000 7f2ba3e00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b880a3cd0 0x7f2b880a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2b94000c00 tx=0x7f2b94005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.990+0000 7f2ba8e00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] conn(0x7f2b880aee30 0x7f2b880b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:41.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.991+0000 7f2ba8e00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] conn(0x7f2b880aee30 0x7f2b880b12d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:41.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:41.992+0000 7f2ba0c00700 1 -- 172.21.15.16:0/4240222347 <== osd.5 v2:172.21.15.188:6808/2068292885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2b90019e50 con 0x7f2b880aee30 2024-04-13T03:32:42.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.011+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 --> [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2ba4063680 con 0x7f2b880aee30 2024-04-13T03:32:42.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.012+0000 7f2ba0c00700 1 -- 172.21.15.16:0/4240222347 <== osd.5 v2:172.21.15.188:6808/2068292885 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2ba4063680 con 0x7f2b880aee30 2024-04-13T03:32:42.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 >> [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] conn(0x7f2b880aee30 msgr2=0x7f2b880b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:42.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] conn(0x7f2b880aee30 0x7f2b880b12d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:42.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b880a3cd0 msgr2=0x7f2b880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:42.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2b880a3cd0 0x7f2b880a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2b94000c00 tx=0x7f2b94005fb0 comp rx=0 tx=0).stop 2024-04-13T03:32:42.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba4109cf0 msgr2=0x7f2ba41a5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:42.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 --2- 172.21.15.16:0/4240222347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ba4109cf0 0x7f2ba41a5be0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000e9f0 tx=0x7f2b9000ed00 comp rx=0 tx=0).stop 2024-04-13T03:32:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2ba8e00700 1 -- 172.21.15.16:0/4240222347 reap_dead start 2024-04-13T03:32:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 shutdown_connections 2024-04-13T03:32:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 >> 172.21.15.16:0/4240222347 conn(0x7f2ba4076280 msgr2=0x7f2ba40785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 shutdown_connections 2024-04-13T03:32:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:42.013+0000 7f2baae00700 1 -- 172.21.15.16:0/4240222347 wait complete. 2024-04-13T03:32:42.063 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-04-13T03:32:42.063 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-13T03:32:42.063 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-13T03:32:42.063 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-13T03:32:42.063 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-13T03:32:42.063 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-13T03:32:42.063 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-13T03:32:42.064 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-04-13T03:32:42.064 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-13T03:32:42.064 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-13T03:32:42.064 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:41 smithi119 ceph-mon[35835]: osdmap e91: 12 total, 12 up, 12 in 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-13T03:32:42.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-13T03:32:42.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:41 smithi188 ceph-mon[36398]: osdmap e91: 12 total, 12 up, 12 in 2024-04-13T03:32:43.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:42 smithi016 ceph-mon[31772]: osdmap e92: 12 total, 12 up, 12 in 2024-04-13T03:32:43.055 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:42 smithi016 ceph-mon[31772]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-13T03:32:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:42 smithi119 ceph-mon[35835]: osdmap e92: 12 total, 12 up, 12 in 2024-04-13T03:32:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:42 smithi119 ceph-mon[35835]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-13T03:32:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:42 smithi188 ceph-mon[36398]: osdmap e92: 12 total, 12 up, 12 in 2024-04-13T03:32:43.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:42 smithi188 ceph-mon[36398]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-13T03:32:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.704+0000 7ff398800700 1 -- 172.21.15.16:0/2299801828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff390108aa0 msgr2=0x7ff390108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.704+0000 7ff398800700 1 --2- 172.21.15.16:0/2299801828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff390108aa0 0x7ff390108e80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff384009990 tx=0x7ff384009ca0 comp rx=0 tx=0).stop 2024-04-13T03:32:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.704+0000 7ff398800700 1 -- 172.21.15.16:0/2299801828 shutdown_connections 2024-04-13T03:32:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.704+0000 7ff398800700 1 --2- 172.21.15.16:0/2299801828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff39006b160 0x7ff39010f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.704+0000 7ff398800700 1 --2- 172.21.15.16:0/2299801828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff390108aa0 0x7ff390108e80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.704+0000 7ff398800700 1 --2- 172.21.15.16:0/2299801828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff39006a7c0 0x7ff39006ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.704+0000 7ff398800700 1 -- 172.21.15.16:0/2299801828 >> 172.21.15.16:0/2299801828 conn(0x7ff390078960 msgr2=0x7ff390078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.705+0000 7ff398800700 1 -- 172.21.15.16:0/2299801828 shutdown_connections 2024-04-13T03:32:43.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.705+0000 7ff398800700 1 -- 172.21.15.16:0/2299801828 wait complete. 2024-04-13T03:32:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.706+0000 7ff398800700 1 Processor -- start 2024-04-13T03:32:43.707 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.706+0000 7ff398800700 1 -- start start 2024-04-13T03:32:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.707+0000 7ff398800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff39006a7c0 0x7ff3901992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.707+0000 7ff398800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff39006b160 0x7ff3901997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.707+0000 7ff398800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff390108aa0 0x7ff39019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.707+0000 7ff398800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff390112d70 con 0x7ff39006a7c0 2024-04-13T03:32:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.707+0000 7ff398800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff390112bf0 con 0x7ff39006b160 2024-04-13T03:32:43.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.707+0000 7ff398800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff390112ef0 con 0x7ff390108aa0 2024-04-13T03:32:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.707+0000 7ff395400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff39006b160 0x7ff3901997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.707+0000 7ff395400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff39006b160 0x7ff3901997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:59536/0 (socket says 172.21.15.16:59536) 2024-04-13T03:32:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.707+0000 7ff395400700 1 -- 172.21.15.16:0/1448720752 learned_addr learned my addr 172.21.15.16:0/1448720752 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.708+0000 7ff396800700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff390108aa0 0x7ff39019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.708+0000 7ff395e00700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff39006a7c0 0x7ff3901992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.708+0000 7ff395400700 1 -- 172.21.15.16:0/1448720752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff390108aa0 msgr2=0x7ff39019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:43.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.708+0000 7ff395400700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff390108aa0 0x7ff39019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.708+0000 7ff395400700 1 -- 172.21.15.16:0/1448720752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff39006a7c0 msgr2=0x7ff3901992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.708+0000 7ff395400700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff39006a7c0 0x7ff3901992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.708+0000 7ff395400700 1 -- 172.21.15.16:0/1448720752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39019e260 con 0x7ff39006b160 2024-04-13T03:32:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.708+0000 7ff395e00700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff39006a7c0 0x7ff3901992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-13T03:32:43.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.708+0000 7ff395400700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff39006b160 0x7ff3901997e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e9a0 tx=0x7ff38000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:43.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.709+0000 7ff38aa00700 1 -- 172.21.15.16:0/1448720752 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380020070 con 0x7ff39006b160 2024-04-13T03:32:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.709+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff39019e550 con 0x7ff39006b160 2024-04-13T03:32:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.709+0000 7ff38aa00700 1 -- 172.21.15.16:0/1448720752 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff38000cca0 con 0x7ff39006b160 2024-04-13T03:32:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.709+0000 7ff38aa00700 1 -- 172.21.15.16:0/1448720752 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38001b980 con 0x7ff39006b160 2024-04-13T03:32:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.709+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3901019b0 con 0x7ff39006b160 2024-04-13T03:32:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.710+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff390063240 con 0x7ff39006b160 2024-04-13T03:32:43.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.713+0000 7ff38aa00700 1 -- 172.21.15.16:0/1448720752 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff380023070 con 0x7ff39006b160 2024-04-13T03:32:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.714+0000 7ff38aa00700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff3780a3d50 0x7ff3780a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:43.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.715+0000 7ff38aa00700 1 -- 172.21.15.16:0/1448720752 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff3800d4a50 con 0x7ff39006b160 2024-04-13T03:32:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.715+0000 7ff395e00700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff3780a3d50 0x7ff3780a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.715+0000 7ff38aa00700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] conn(0x7ff3780aeeb0 0x7ff3780b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.715+0000 7ff38aa00700 1 -- 172.21.15.16:0/1448720752 --> [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff38000ce50 con 0x7ff3780aeeb0 2024-04-13T03:32:43.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.715+0000 7ff38aa00700 1 -- 172.21.15.16:0/1448720752 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7ff3800d4e00 con 0x7ff39006b160 2024-04-13T03:32:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.715+0000 7ff396800700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] conn(0x7ff3780aeeb0 0x7ff3780b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.716+0000 7ff395e00700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff3780a3d50 0x7ff3780a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff38400b2e0 tx=0x7ff384005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:43.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.716+0000 7ff396800700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] conn(0x7ff3780aeeb0 0x7ff3780b1350 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:43.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.717+0000 7ff38aa00700 1 -- 172.21.15.16:0/1448720752 <== osd.11 v2:172.21.15.188:6824/2606023375 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff38000ce50 con 0x7ff3780aeeb0 2024-04-13T03:32:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.735+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 --> [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff390063640 con 0x7ff3780aeeb0 2024-04-13T03:32:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.736+0000 7ff38aa00700 1 -- 172.21.15.16:0/1448720752 <== osd.11 v2:172.21.15.188:6824/2606023375 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff390063640 con 0x7ff3780aeeb0 2024-04-13T03:32:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 >> [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] conn(0x7ff3780aeeb0 msgr2=0x7ff3780b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] conn(0x7ff3780aeeb0 0x7ff3780b1350 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff3780a3d50 msgr2=0x7ff3780a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff3780a3d50 0x7ff3780a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff38400b2e0 tx=0x7ff384005e80 comp rx=0 tx=0).stop 2024-04-13T03:32:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff39006b160 msgr2=0x7ff3901997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 --2- 172.21.15.16:0/1448720752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff39006b160 0x7ff3901997e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e9a0 tx=0x7ff38000ecb0 comp rx=0 tx=0).stop 2024-04-13T03:32:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff396800700 1 -- 172.21.15.16:0/1448720752 reap_dead start 2024-04-13T03:32:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 shutdown_connections 2024-04-13T03:32:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 >> 172.21.15.16:0/1448720752 conn(0x7ff390078960 msgr2=0x7ff3900ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 shutdown_connections 2024-04-13T03:32:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.737+0000 7ff398800700 1 -- 172.21.15.16:0/1448720752 wait complete. 2024-04-13T03:32:43.900 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:43 smithi016 ceph-mon[31772]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-13T03:32:44.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:43 smithi119 ceph-mon[35835]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-13T03:32:44.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:43 smithi188 ceph-mon[36398]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-13T03:32:44.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.609+0000 7f4760a00700 1 -- 172.21.15.16:0/1951142515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f475c104f50 msgr2=0x7f475c105330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.609+0000 7f4760a00700 1 --2- 172.21.15.16:0/1951142515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f475c104f50 0x7f475c105330 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f474c0099c0 tx=0x7f474c009cd0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.609+0000 7f4760a00700 1 -- 172.21.15.16:0/1951142515 shutdown_connections 2024-04-13T03:32:44.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.609+0000 7f4760a00700 1 --2- 172.21.15.16:0/1951142515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f475c10a030 0x7f475c111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.609+0000 7f4760a00700 1 --2- 172.21.15.16:0/1951142515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f475c104f50 0x7f475c105330 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.609+0000 7f4760a00700 1 --2- 172.21.15.16:0/1951142515 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f475c105900 0x7f475c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.609+0000 7f4760a00700 1 -- 172.21.15.16:0/1951142515 >> 172.21.15.16:0/1951142515 conn(0x7f475c1007e0 msgr2=0x7f475c102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:44.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.609+0000 7f4760a00700 1 -- 172.21.15.16:0/1951142515 shutdown_connections 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.609+0000 7f4760a00700 1 -- 172.21.15.16:0/1951142515 wait complete. 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.610+0000 7f4760a00700 1 Processor -- start 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.610+0000 7f4760a00700 1 -- start start 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4760a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f475c104f50 0x7f475c19b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4760a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f475c105900 0x7f475c19b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4760a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f475c10a030 0x7f475c19fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4760a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f475c115010 con 0x7f475c105900 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4760a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f475c114e90 con 0x7f475c10a030 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4760a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f475c115190 con 0x7f475c104f50 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4759400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f475c105900 0x7f475c19b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4759400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f475c105900 0x7f475c19b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55512/0 (socket says 172.21.15.16:55512) 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4759400700 1 -- 172.21.15.16:0/3067007129 learned_addr learned my addr 172.21.15.16:0/3067007129 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f4759e00700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f475c104f50 0x7f475c19b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.611+0000 7f475a800700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f475c10a030 0x7f475c19fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.612+0000 7f4759400700 1 -- 172.21.15.16:0/3067007129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f475c104f50 msgr2=0x7f475c19b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.612+0000 7f4759400700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f475c104f50 0x7f475c19b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.612+0000 7f4759400700 1 -- 172.21.15.16:0/3067007129 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f475c10a030 msgr2=0x7f475c19fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.612+0000 7f4759400700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f475c10a030 0x7f475c19fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.612+0000 7f4759400700 1 -- 172.21.15.16:0/3067007129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475c1a0470 con 0x7f475c105900 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.612+0000 7f4759400700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f475c105900 0x7f475c19b9a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f474800e970 tx=0x7f474800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.612+0000 7f4752a00700 1 -- 172.21.15.16:0/3067007129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748004a30 con 0x7f475c105900 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.612+0000 7f4752a00700 1 -- 172.21.15.16:0/3067007129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f474800ccc0 con 0x7f475c105900 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.612+0000 7f4752a00700 1 -- 172.21.15.16:0/3067007129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748005650 con 0x7f475c105900 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.613+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f475c1a0760 con 0x7f475c105900 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.613+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f475c1a7fe0 con 0x7f475c105900 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.614+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f475c063200 con 0x7f475c105900 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.617+0000 7f4752a00700 1 -- 172.21.15.16:0/3067007129 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4748004bd0 con 0x7f475c105900 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.618+0000 7f4752a00700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f47400a3d50 0x7f47400a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.618+0000 7f4752a00700 1 -- 172.21.15.16:0/3067007129 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f47480d1270 con 0x7f475c105900 2024-04-13T03:32:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.618+0000 7f4759e00700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f47400a3d50 0x7f47400a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.619+0000 7f4752a00700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] conn(0x7f47400aeeb0 0x7f47400b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.619+0000 7f4752a00700 1 -- 172.21.15.16:0/3067007129 --> [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f47400b1a20 con 0x7f47400aeeb0 2024-04-13T03:32:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.619+0000 7f4752a00700 1 -- 172.21.15.16:0/3067007129 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f47480d1620 con 0x7f475c105900 2024-04-13T03:32:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.619+0000 7f475a800700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] conn(0x7f47400aeeb0 0x7f47400b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.619+0000 7f4759e00700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f47400a3d50 0x7f47400a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f474c000c00 tx=0x7f474c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:44.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.620+0000 7f475a800700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] conn(0x7f47400aeeb0 0x7f47400b1350 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.621+0000 7f4752a00700 1 -- 172.21.15.16:0/3067007129 <== osd.7 v2:172.21.15.119:6816/2208117973 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f47400b1a20 con 0x7f47400aeeb0 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.639+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 --> [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f475c0636d0 con 0x7f47400aeeb0 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.640+0000 7f4752a00700 1 -- 172.21.15.16:0/3067007129 <== osd.7 v2:172.21.15.119:6816/2208117973 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f475c0636d0 con 0x7f47400aeeb0 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.640+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 >> [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] conn(0x7f47400aeeb0 msgr2=0x7f47400b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.640+0000 7f4760a00700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] conn(0x7f47400aeeb0 0x7f47400b1350 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.641+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f47400a3d50 msgr2=0x7f47400a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.641+0000 7f4760a00700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f47400a3d50 0x7f47400a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f474c000c00 tx=0x7f474c005fb0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.641+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f475c105900 msgr2=0x7f475c19b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.641+0000 7f4760a00700 1 --2- 172.21.15.16:0/3067007129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f475c105900 0x7f475c19b9a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f474800e970 tx=0x7f474800ec80 comp rx=0 tx=0).stop 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.641+0000 7f475a800700 1 -- 172.21.15.16:0/3067007129 reap_dead start 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.641+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 shutdown_connections 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.641+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 >> 172.21.15.16:0/3067007129 conn(0x7f475c1007e0 msgr2=0x7f475c102580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.641+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 shutdown_connections 2024-04-13T03:32:44.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:43.641+0000 7f4760a00700 1 -- 172.21.15.16:0/3067007129 wait complete. 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.588+0000 7ff795800700 1 -- 172.21.15.16:0/3268996862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff790074ce0 msgr2=0x7ff7900750c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.588+0000 7ff795800700 1 --2- 172.21.15.16:0/3268996862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff790074ce0 0x7ff7900750c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff780009990 tx=0x7ff780009ca0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.588+0000 7ff795800700 1 -- 172.21.15.16:0/3268996862 shutdown_connections 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.588+0000 7ff795800700 1 --2- 172.21.15.16:0/3268996862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff79010f4b0 0x7ff7901118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.588+0000 7ff795800700 1 --2- 172.21.15.16:0/3268996862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff790074ce0 0x7ff7900750c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.588+0000 7ff795800700 1 --2- 172.21.15.16:0/3268996862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff790075600 0x7ff79010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.588+0000 7ff795800700 1 -- 172.21.15.16:0/3268996862 >> 172.21.15.16:0/3268996862 conn(0x7ff790079c10 msgr2=0x7ff79007a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.588+0000 7ff795800700 1 -- 172.21.15.16:0/3268996862 shutdown_connections 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.588+0000 7ff795800700 1 -- 172.21.15.16:0/3268996862 wait complete. 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.590+0000 7ff795800700 1 Processor -- start 2024-04-13T03:32:44.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.590+0000 7ff795800700 1 -- start start 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.590+0000 7ff795800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff790074ce0 0x7ff79019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.590+0000 7ff795800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff790075600 0x7ff79019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.590+0000 7ff795800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff79010f4b0 0x7ff79019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.590+0000 7ff795800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff790115030 con 0x7ff790075600 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.590+0000 7ff795800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff790114eb0 con 0x7ff790074ce0 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.590+0000 7ff795800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff7901151b0 con 0x7ff79010f4b0 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff790074ce0 0x7ff79019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff790074ce0 0x7ff79019b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:59452/0 (socket says 172.21.15.16:59452) 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78ea00700 1 -- 172.21.15.16:0/2760121267 learned_addr learned my addr 172.21.15.16:0/2760121267 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78e000700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff790075600 0x7ff79019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78f400700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff79010f4b0 0x7ff79019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78ea00700 1 -- 172.21.15.16:0/2760121267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff79010f4b0 msgr2=0x7ff79019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78ea00700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff79010f4b0 0x7ff79019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78ea00700 1 -- 172.21.15.16:0/2760121267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff790075600 msgr2=0x7ff79019ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78ea00700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff790075600 0x7ff79019ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78ea00700 1 -- 172.21.15.16:0/2760121267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7901a04c0 con 0x7ff790074ce0 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78f400700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff79010f4b0 0x7ff79019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.591+0000 7ff78ea00700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff790074ce0 0x7ff79019b500 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff780000c00 tx=0x7ff78000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.592+0000 7ff787400700 1 -- 172.21.15.16:0/2760121267 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780027070 con 0x7ff790074ce0 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.592+0000 7ff787400700 1 -- 172.21.15.16:0/2760121267 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff780003d70 con 0x7ff790074ce0 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.592+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7901a0750 con 0x7ff790074ce0 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.592+0000 7ff787400700 1 -- 172.21.15.16:0/2760121267 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78000f460 con 0x7ff790074ce0 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.592+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7901a8030 con 0x7ff790074ce0 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.593+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff7901018f0 con 0x7ff790074ce0 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.595+0000 7ff787400700 1 -- 172.21.15.16:0/2760121267 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff78002a070 con 0x7ff790074ce0 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.597+0000 7ff787400700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff7740a3cd0 0x7ff7740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.597+0000 7ff787400700 1 -- 172.21.15.16:0/2760121267 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff7800d1eb0 con 0x7ff790074ce0 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.598+0000 7ff78e000700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff7740a3cd0 0x7ff7740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.598+0000 7ff787400700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] conn(0x7ff7740ae990 0x7ff7740b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.598+0000 7ff787400700 1 -- 172.21.15.16:0/2760121267 --> [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff780003bc0 con 0x7ff7740ae990 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.598+0000 7ff787400700 1 -- 172.21.15.16:0/2760121267 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff7800d2260 con 0x7ff790074ce0 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.598+0000 7ff78f400700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] conn(0x7ff7740ae990 0x7ff7740b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.598+0000 7ff78e000700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff7740a3cd0 0x7ff7740a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff790111070 tx=0x7ff77c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.599+0000 7ff78f400700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] conn(0x7ff7740ae990 0x7ff7740b0dd0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.600+0000 7ff787400700 1 -- 172.21.15.16:0/2760121267 <== osd.2 v2:172.21.15.188:6800/1923423179 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff780003bc0 con 0x7ff7740ae990 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.618+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 --> [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff7900763d0 con 0x7ff7740ae990 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.618+0000 7ff787400700 1 -- 172.21.15.16:0/2760121267 <== osd.2 v2:172.21.15.188:6800/1923423179 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff7900763d0 con 0x7ff7740ae990 2024-04-13T03:32:44.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.619+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 >> [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] conn(0x7ff7740ae990 msgr2=0x7ff7740b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.619+0000 7ff795800700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] conn(0x7ff7740ae990 0x7ff7740b0dd0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.619+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff7740a3cd0 msgr2=0x7ff7740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.619+0000 7ff795800700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff7740a3cd0 0x7ff7740a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff790111070 tx=0x7ff77c009260 comp rx=0 tx=0).stop 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.619+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff790074ce0 msgr2=0x7ff79019b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.619+0000 7ff795800700 1 --2- 172.21.15.16:0/2760121267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff790074ce0 0x7ff79019b500 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff780000c00 tx=0x7ff78000bab0 comp rx=0 tx=0).stop 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.619+0000 7ff78f400700 1 -- 172.21.15.16:0/2760121267 reap_dead start 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.619+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 shutdown_connections 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.619+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 >> 172.21.15.16:0/2760121267 conn(0x7ff790079c10 msgr2=0x7ff790100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.620+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 shutdown_connections 2024-04-13T03:32:44.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:39.620+0000 7ff795800700 1 -- 172.21.15.16:0/2760121267 wait complete. 2024-04-13T03:32:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:44 smithi119 ceph-mon[35835]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 9 op/s; 13 B/s, 0 objects/s recovering 2024-04-13T03:32:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:44 smithi016 ceph-mon[31772]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 9 op/s; 13 B/s, 0 objects/s recovering 2024-04-13T03:32:45.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:44 smithi188 ceph-mon[36398]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 9 op/s; 13 B/s, 0 objects/s recovering 2024-04-13T03:32:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.096+0000 7ff0ad400700 1 -- 172.21.15.16:0/1888700319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0a8077900 msgr2=0x7ff0a8113320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.096+0000 7ff0ad400700 1 --2- 172.21.15.16:0/1888700319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0a8077900 0x7ff0a8113320 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff094009990 tx=0x7ff094009ca0 comp rx=0 tx=0).stop 2024-04-13T03:32:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.096+0000 7ff0ad400700 1 -- 172.21.15.16:0/1888700319 shutdown_connections 2024-04-13T03:32:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.096+0000 7ff0ad400700 1 --2- 172.21.15.16:0/1888700319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a8113860 0x7ff0a8115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.096+0000 7ff0ad400700 1 --2- 172.21.15.16:0/1888700319 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a8076fe0 0x7ff0a80773c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.096+0000 7ff0ad400700 1 --2- 172.21.15.16:0/1888700319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0a8077900 0x7ff0a8113320 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:46.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.096+0000 7ff0ad400700 1 -- 172.21.15.16:0/1888700319 >> 172.21.15.16:0/1888700319 conn(0x7ff0a81006f0 msgr2=0x7ff0a8102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.096+0000 7ff0ad400700 1 -- 172.21.15.16:0/1888700319 shutdown_connections 2024-04-13T03:32:46.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.096+0000 7ff0ad400700 1 -- 172.21.15.16:0/1888700319 wait complete. 2024-04-13T03:32:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.097+0000 7ff0ad400700 1 Processor -- start 2024-04-13T03:32:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0ad400700 1 -- start start 2024-04-13T03:32:46.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0ad400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a8076fe0 0x7ff0a81a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0ad400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a8077900 0x7ff0a81a5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0ad400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0a8113860 0x7ff0a819f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0ad400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a8119340 con 0x7ff0a8077900 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0ad400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a81191c0 con 0x7ff0a8076fe0 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0ad400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a81194c0 con 0x7ff0a8113860 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0a7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a8077900 0x7ff0a81a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0a7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a8077900 0x7ff0a81a5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55550/0 (socket says 172.21.15.16:55550) 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0a7e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a8076fe0 0x7ff0a81a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.098+0000 7ff0a7400700 1 -- 172.21.15.16:0/121352847 learned_addr learned my addr 172.21.15.16:0/121352847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.099+0000 7ff0aca00700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0a8113860 0x7ff0a819f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.099+0000 7ff0a7e00700 1 -- 172.21.15.16:0/121352847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0a8113860 msgr2=0x7ff0a819f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:46.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.099+0000 7ff0a7e00700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff0a8113860 0x7ff0a819f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.099+0000 7ff0a7e00700 1 -- 172.21.15.16:0/121352847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a8077900 msgr2=0x7ff0a81a5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.099+0000 7ff0a7e00700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a8077900 0x7ff0a81a5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.099+0000 7ff0a7e00700 1 -- 172.21.15.16:0/121352847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a819fd20 con 0x7ff0a8076fe0 2024-04-13T03:32:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.099+0000 7ff0a7400700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a8077900 0x7ff0a81a5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:46.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.099+0000 7ff0a7e00700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a8076fe0 0x7ff0a81a5400 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff09800e9f0 tx=0x7ff09800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.100+0000 7ff0a4c00700 1 -- 172.21.15.16:0/121352847 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09800c9c0 con 0x7ff0a8076fe0 2024-04-13T03:32:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.100+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a81a0010 con 0x7ff0a8076fe0 2024-04-13T03:32:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.100+0000 7ff0a4c00700 1 -- 172.21.15.16:0/121352847 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff09800cb60 con 0x7ff0a8076fe0 2024-04-13T03:32:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.100+0000 7ff0a4c00700 1 -- 172.21.15.16:0/121352847 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098018680 con 0x7ff0a8076fe0 2024-04-13T03:32:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.100+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a81ac2d0 con 0x7ff0a8076fe0 2024-04-13T03:32:46.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.101+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff0a8063240 con 0x7ff0a8076fe0 2024-04-13T03:32:46.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.103+0000 7ff0a4c00700 1 -- 172.21.15.16:0/121352847 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff09800cd10 con 0x7ff0a8076fe0 2024-04-13T03:32:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.104+0000 7ff0a4c00700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff08c0a3d50 0x7ff08c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.104+0000 7ff0a4c00700 1 -- 172.21.15.16:0/121352847 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7ff0980d0da0 con 0x7ff0a8076fe0 2024-04-13T03:32:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.105+0000 7ff0a7400700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff08c0a3d50 0x7ff08c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.105+0000 7ff0a4c00700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] conn(0x7ff08c0aeeb0 0x7ff08c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.105+0000 7ff0a4c00700 1 -- 172.21.15.16:0/121352847 --> [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff098021e50 con 0x7ff08c0aeeb0 2024-04-13T03:32:46.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.105+0000 7ff0a4c00700 1 -- 172.21.15.16:0/121352847 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7ff0980d1150 con 0x7ff0a8076fe0 2024-04-13T03:32:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.105+0000 7ff0aca00700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] conn(0x7ff08c0aeeb0 0x7ff08c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.106+0000 7ff0a7400700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff08c0a3d50 0x7ff08c0a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8104160 tx=0x7ff094009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.106+0000 7ff0aca00700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] conn(0x7ff08c0aeeb0 0x7ff08c0b1350 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.107+0000 7ff0a4c00700 1 -- 172.21.15.16:0/121352847 <== osd.10 v2:172.21.15.119:6824/1724485593 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff098021e50 con 0x7ff08c0aeeb0 2024-04-13T03:32:46.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.125+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 --> [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff0a80636c0 con 0x7ff08c0aeeb0 2024-04-13T03:32:46.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.126+0000 7ff0a4c00700 1 -- 172.21.15.16:0/121352847 <== osd.10 v2:172.21.15.119:6824/1724485593 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff0a80636c0 con 0x7ff08c0aeeb0 2024-04-13T03:32:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.126+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 >> [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] conn(0x7ff08c0aeeb0 msgr2=0x7ff08c0b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.126+0000 7ff0ad400700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] conn(0x7ff08c0aeeb0 0x7ff08c0b1350 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.127+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff08c0a3d50 msgr2=0x7ff08c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.127+0000 7ff0ad400700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff08c0a3d50 0x7ff08c0a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8104160 tx=0x7ff094009590 comp rx=0 tx=0).stop 2024-04-13T03:32:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.127+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a8076fe0 msgr2=0x7ff0a81a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.127+0000 7ff0ad400700 1 --2- 172.21.15.16:0/121352847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a8076fe0 0x7ff0a81a5400 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff09800e9f0 tx=0x7ff09800ed00 comp rx=0 tx=0).stop 2024-04-13T03:32:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.127+0000 7ff0aca00700 1 -- 172.21.15.16:0/121352847 reap_dead start 2024-04-13T03:32:46.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.127+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 shutdown_connections 2024-04-13T03:32:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.127+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 >> 172.21.15.16:0/121352847 conn(0x7ff0a81006f0 msgr2=0x7ff0a8102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.127+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 shutdown_connections 2024-04-13T03:32:46.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.127+0000 7ff0ad400700 1 -- 172.21.15.16:0/121352847 wait complete. 2024-04-13T03:32:47.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:46 smithi016 ceph-mon[31772]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 9 op/s; 7.8 KiB/s, 1 objects/s recovering 2024-04-13T03:32:47.240 INFO:teuthology.orchestra.run.smithi016.stdout:77309411407 2024-04-13T03:32:47.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-13T03:32:47.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:46 smithi119 ceph-mon[35835]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 9 op/s; 7.8 KiB/s, 1 objects/s recovering 2024-04-13T03:32:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:46 smithi188 ceph-mon[36398]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 9 op/s; 7.8 KiB/s, 1 objects/s recovering 2024-04-13T03:32:47.791 INFO:teuthology.orchestra.run.smithi016.stdout:219043332147 2024-04-13T03:32:47.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-13T03:32:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.728+0000 7f0ced200700 1 -- 172.21.15.16:0/70905000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce806b160 msgr2=0x7f0ce810f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.728+0000 7f0ced200700 1 --2- 172.21.15.16:0/70905000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce806b160 0x7f0ce810f970 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00098f0 tx=0x7f0ce0009c00 comp rx=0 tx=0).stop 2024-04-13T03:32:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.729+0000 7f0ced200700 1 -- 172.21.15.16:0/70905000 shutdown_connections 2024-04-13T03:32:47.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.729+0000 7f0ced200700 1 --2- 172.21.15.16:0/70905000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce806b160 0x7f0ce810f970 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.729+0000 7f0ced200700 1 --2- 172.21.15.16:0/70905000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce8108aa0 0x7f0ce8108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.729+0000 7f0ced200700 1 --2- 172.21.15.16:0/70905000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce806a7c0 0x7f0ce806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.729+0000 7f0ced200700 1 -- 172.21.15.16:0/70905000 >> 172.21.15.16:0/70905000 conn(0x7f0ce8078960 msgr2=0x7f0ce8078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.729+0000 7f0ced200700 1 -- 172.21.15.16:0/70905000 shutdown_connections 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.729+0000 7f0ced200700 1 -- 172.21.15.16:0/70905000 wait complete. 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.730+0000 7f0ced200700 1 Processor -- start 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.730+0000 7f0ced200700 1 -- start start 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ced200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce806a7c0 0x7f0ce8101a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ced200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce806b160 0x7f0ce8101fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ced200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce8108aa0 0x7f0ce8102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ced200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce8112de0 con 0x7f0ce806b160 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ced200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce8112c60 con 0x7f0ce8108aa0 2024-04-13T03:32:47.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ced200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce8112f60 con 0x7f0ce806a7c0 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0cdd400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce806b160 0x7f0ce8101fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ce5e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce806a7c0 0x7f0ce8101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ce5e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce806a7c0 0x7f0ce8101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:39668/0 (socket says 172.21.15.16:39668) 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ce5e00700 1 -- 172.21.15.16:0/1175467358 learned_addr learned my addr 172.21.15.16:0/1175467358 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.731+0000 7f0ce6800700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce8108aa0 0x7f0ce8102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.732+0000 7f0ce5e00700 1 -- 172.21.15.16:0/1175467358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce8108aa0 msgr2=0x7f0ce8102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.732+0000 7f0ce5e00700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce8108aa0 0x7f0ce8102500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.732+0000 7f0ce5e00700 1 -- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce806b160 msgr2=0x7f0ce8101fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.732+0000 7f0ce5e00700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce806b160 0x7f0ce8101fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.732+0000 7f0ce5e00700 1 -- 172.21.15.16:0/1175467358 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce81abc00 con 0x7f0ce806a7c0 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.732+0000 7f0cdd400700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ce806b160 0x7f0ce8101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.732+0000 7f0ce5e00700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce806a7c0 0x7f0ce8101a80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0cd800e3c0 tx=0x7f0cd800e6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.732+0000 7f0cdf400700 1 -- 172.21.15.16:0/1175467358 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd800ee40 con 0x7f0ce806a7c0 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.733+0000 7f0cdf400700 1 -- 172.21.15.16:0/1175467358 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd8003a00 con 0x7f0ce806a7c0 2024-04-13T03:32:47.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.733+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce81abe90 con 0x7f0ce806a7c0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.733+0000 7f0cdf400700 1 -- 172.21.15.16:0/1175467358 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8004790 con 0x7f0ce806a7c0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.733+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce81ac2e0 con 0x7f0ce806a7c0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.734+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ce80631c0 con 0x7f0ce806a7c0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.736+0000 7f0cdf400700 1 -- 172.21.15.16:0/1175467358 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0cd801b030 con 0x7f0ce806a7c0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.737+0000 7f0cdf400700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ccc0a3cd0 0x7f0ccc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.738+0000 7f0cdf400700 1 -- 172.21.15.16:0/1175467358 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f0cd80d0a90 con 0x7f0ce806a7c0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.738+0000 7f0cdd400700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ccc0a3cd0 0x7f0ccc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.738+0000 7f0cdf400700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] conn(0x7f0ccc0aee30 0x7f0ccc0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.738+0000 7f0cdf400700 1 -- 172.21.15.16:0/1175467358 --> [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0cd8024ea0 con 0x7f0ccc0aee30 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.738+0000 7f0cdf400700 1 -- 172.21.15.16:0/1175467358 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f0cd80d0e40 con 0x7f0ce806a7c0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.738+0000 7f0ce6800700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] conn(0x7f0ccc0aee30 0x7f0ccc0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.738+0000 7f0cdd400700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ccc0a3cd0 0x7f0ccc0a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8103240 tx=0x7f0cd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:47.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.739+0000 7f0ce6800700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] conn(0x7f0ccc0aee30 0x7f0ccc0b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.740+0000 7f0cdf400700 1 -- 172.21.15.16:0/1175467358 <== osd.3 v2:172.21.15.16:6810/3993319760 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0cd8024ea0 con 0x7f0ccc0aee30 2024-04-13T03:32:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.759+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 --> [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ce8063640 con 0x7f0ccc0aee30 2024-04-13T03:32:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.759+0000 7f0cdf400700 1 -- 172.21.15.16:0/1175467358 <== osd.3 v2:172.21.15.16:6810/3993319760 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0ce8063640 con 0x7f0ccc0aee30 2024-04-13T03:32:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] conn(0x7f0ccc0aee30 msgr2=0x7f0ccc0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] conn(0x7f0ccc0aee30 0x7f0ccc0b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:47.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ccc0a3cd0 msgr2=0x7f0ccc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ccc0a3cd0 0x7f0ccc0a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8103240 tx=0x7f0cd0009260 comp rx=0 tx=0).stop 2024-04-13T03:32:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce806a7c0 msgr2=0x7f0ce8101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 --2- 172.21.15.16:0/1175467358 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ce806a7c0 0x7f0ce8101a80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0cd800e3c0 tx=0x7f0cd800e6d0 comp rx=0 tx=0).stop 2024-04-13T03:32:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ce6800700 1 -- 172.21.15.16:0/1175467358 reap_dead start 2024-04-13T03:32:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 shutdown_connections 2024-04-13T03:32:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 >> 172.21.15.16:0/1175467358 conn(0x7f0ce8078960 msgr2=0x7f0ce80ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 shutdown_connections 2024-04-13T03:32:47.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:46.760+0000 7f0ced200700 1 -- 172.21.15.16:0/1175467358 wait complete. 2024-04-13T03:32:47.988 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:47 smithi016 ceph-mon[31772]: Health check update: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-13T03:32:48.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:47 smithi119 ceph-mon[35835]: Health check update: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-13T03:32:48.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:47 smithi188 ceph-mon[36398]: Health check update: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-13T03:32:49.110 INFO:teuthology.orchestra.run.smithi016.stdout:347892351003 2024-04-13T03:32:49.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-13T03:32:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.124+0000 7f80e4e00700 1 -- 172.21.15.16:0/3065551015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e0077900 msgr2=0x7f80e0113430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.124+0000 7f80e4e00700 1 --2- 172.21.15.16:0/3065551015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e0077900 0x7f80e0113430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009990 tx=0x7f80c8009ca0 comp rx=0 tx=0).stop 2024-04-13T03:32:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.124+0000 7f80e4e00700 1 -- 172.21.15.16:0/3065551015 shutdown_connections 2024-04-13T03:32:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.124+0000 7f80e4e00700 1 --2- 172.21.15.16:0/3065551015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e0113970 0x7f80e0115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.124+0000 7f80e4e00700 1 --2- 172.21.15.16:0/3065551015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80e0076fe0 0x7f80e00773c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.124+0000 7f80e4e00700 1 --2- 172.21.15.16:0/3065551015 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e0077900 0x7f80e0113430 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.124+0000 7f80e4e00700 1 -- 172.21.15.16:0/3065551015 >> 172.21.15.16:0/3065551015 conn(0x7f80e01007c0 msgr2=0x7f80e0102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.124+0000 7f80e4e00700 1 -- 172.21.15.16:0/3065551015 shutdown_connections 2024-04-13T03:32:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.125+0000 7f80e4e00700 1 -- 172.21.15.16:0/3065551015 wait complete. 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.126+0000 7f80e4e00700 1 Processor -- start 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.126+0000 7f80e4e00700 1 -- start start 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.126+0000 7f80e4e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e0076fe0 0x7f80e01a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80e4e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e0077900 0x7f80e01a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80e4e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80e0113970 0x7f80e019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80e4e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80e01194a0 con 0x7f80e0076fe0 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80e4e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f80e0119320 con 0x7f80e0113970 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80e4e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f80e0119620 con 0x7f80e0077900 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80dd400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e0077900 0x7f80e01a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80dd400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e0077900 0x7f80e01a5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:38100/0 (socket says 172.21.15.16:38100) 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80dd400700 1 -- 172.21.15.16:0/177656156 learned_addr learned my addr 172.21.15.16:0/177656156 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80dde00700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e0076fe0 0x7f80e01a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80de800700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80e0113970 0x7f80e019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80dd400700 1 -- 172.21.15.16:0/177656156 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80e0113970 msgr2=0x7f80e019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80dd400700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80e0113970 0x7f80e019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80dd400700 1 -- 172.21.15.16:0/177656156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e0076fe0 msgr2=0x7f80e01a5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.127+0000 7f80dd400700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80e0076fe0 0x7f80e01a5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.128+0000 7f80dd400700 1 -- 172.21.15.16:0/177656156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e01a0000 con 0x7f80e0077900 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.128+0000 7f80dd400700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e0077900 0x7f80e01a5b90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f80c8000c00 tx=0x7f80c8003890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.128+0000 7f80d6a00700 1 -- 172.21.15.16:0/177656156 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8027070 con 0x7f80e0077900 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.128+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e01a0290 con 0x7f80e0077900 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.128+0000 7f80d6a00700 1 -- 172.21.15.16:0/177656156 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c8003e40 con 0x7f80e0077900 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.128+0000 7f80d6a00700 1 -- 172.21.15.16:0/177656156 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8022c90 con 0x7f80e0077900 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.129+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e01ac5d0 con 0x7f80e0077900 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.129+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f80e0063200 con 0x7f80e0077900 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.132+0000 7f80d6a00700 1 -- 172.21.15.16:0/177656156 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f80c802a070 con 0x7f80e0077900 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.133+0000 7f80d6a00700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80c40a3cd0 0x7f80c40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.133+0000 7f80d6a00700 1 -- 172.21.15.16:0/177656156 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f80c80d21c0 con 0x7f80e0077900 2024-04-13T03:32:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.134+0000 7f80dde00700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80c40a3cd0 0x7f80c40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.134+0000 7f80d6a00700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] conn(0x7f80c40aee30 0x7f80c40b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.134+0000 7f80d6a00700 1 -- 172.21.15.16:0/177656156 --> [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80c8027e70 con 0x7f80c40aee30 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.134+0000 7f80d6a00700 1 -- 172.21.15.16:0/177656156 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f80c80d2570 con 0x7f80e0077900 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.134+0000 7f80de800700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] conn(0x7f80c40aee30 0x7f80c40b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.134+0000 7f80dde00700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80c40a3cd0 0x7f80c40a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f80d0007be0 tx=0x7f80d00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.135+0000 7f80de800700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] conn(0x7f80c40aee30 0x7f80c40b12d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.136+0000 7f80d6a00700 1 -- 172.21.15.16:0/177656156 <== osd.4 v2:172.21.15.119:6808/1038380555 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80c8027e70 con 0x7f80c40aee30 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.154+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 --> [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f80e0063600 con 0x7f80c40aee30 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.155+0000 7f80d6a00700 1 -- 172.21.15.16:0/177656156 <== osd.4 v2:172.21.15.119:6808/1038380555 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f80e0063600 con 0x7f80c40aee30 2024-04-13T03:32:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.155+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 >> [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] conn(0x7f80c40aee30 msgr2=0x7f80c40b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.155+0000 7f80e4e00700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] conn(0x7f80c40aee30 0x7f80c40b12d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.155+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80c40a3cd0 msgr2=0x7f80c40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.156+0000 7f80e4e00700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80c40a3cd0 0x7f80c40a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f80d0007be0 tx=0x7f80d00073d0 comp rx=0 tx=0).stop 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.156+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e0077900 msgr2=0x7f80e01a5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.156+0000 7f80e4e00700 1 --2- 172.21.15.16:0/177656156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80e0077900 0x7f80e01a5b90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f80c8000c00 tx=0x7f80c8003890 comp rx=0 tx=0).stop 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.156+0000 7f80de800700 1 -- 172.21.15.16:0/177656156 reap_dead start 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.156+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 shutdown_connections 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.156+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 >> 172.21.15.16:0/177656156 conn(0x7f80e01007c0 msgr2=0x7f80e0102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.156+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 shutdown_connections 2024-04-13T03:32:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:49.156+0000 7f80e4e00700 1 -- 172.21.15.16:0/177656156 wait complete. 2024-04-13T03:32:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:48 smithi119 ceph-mon[35835]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 7.8 KiB/s, 1 objects/s recovering 2024-04-13T03:32:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:48 smithi119 ceph-mon[35835]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-04-13T03:32:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:48 smithi119 ceph-mon[35835]: Cluster is now healthy 2024-04-13T03:32:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:48 smithi016 ceph-mon[31772]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 7.8 KiB/s, 1 objects/s recovering 2024-04-13T03:32:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:48 smithi016 ceph-mon[31772]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-04-13T03:32:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:48 smithi016 ceph-mon[31772]: Cluster is now healthy 2024-04-13T03:32:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:48 smithi188 ceph-mon[36398]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 7.8 KiB/s, 1 objects/s recovering 2024-04-13T03:32:49.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:48 smithi188 ceph-mon[36398]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-04-13T03:32:49.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:48 smithi188 ceph-mon[36398]: Cluster is now healthy 2024-04-13T03:32:50.163 INFO:teuthology.orchestra.run.smithi016.stdout:107374182473 2024-04-13T03:32:50.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-13T03:32:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.427+0000 7fb0a2e00700 1 -- 172.21.15.16:0/767011532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c107a00 msgr2=0x7fb09c107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.427+0000 7fb0a2e00700 1 --2- 172.21.15.16:0/767011532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c107a00 0x7fb09c107de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009990 tx=0x7fb08c009ca0 comp rx=0 tx=0).stop 2024-04-13T03:32:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.427+0000 7fb0a2e00700 1 -- 172.21.15.16:0/767011532 shutdown_connections 2024-04-13T03:32:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.427+0000 7fb0a2e00700 1 --2- 172.21.15.16:0/767011532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb09c102370 0x7fb09c10e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:50.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.427+0000 7fb0a2e00700 1 --2- 172.21.15.16:0/767011532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c107a00 0x7fb09c107de0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.427+0000 7fb0a2e00700 1 --2- 172.21.15.16:0/767011532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb09c1019b0 0x7fb09c101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.427+0000 7fb0a2e00700 1 -- 172.21.15.16:0/767011532 >> 172.21.15.16:0/767011532 conn(0x7fb09c078960 msgr2=0x7fb09c078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.427+0000 7fb0a2e00700 1 -- 172.21.15.16:0/767011532 shutdown_connections 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.427+0000 7fb0a2e00700 1 -- 172.21.15.16:0/767011532 wait complete. 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.429+0000 7fb0a2e00700 1 Processor -- start 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.429+0000 7fb0a2e00700 1 -- start start 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.429+0000 7fb0a2e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c1019b0 0x7fb09c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.429+0000 7fb0a2e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb09c102370 0x7fb09c19b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.429+0000 7fb0a2e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb09c107a00 0x7fb09c19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.429+0000 7fb0a2e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c111ca0 con 0x7fb09c1019b0 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.429+0000 7fb0a2e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c111b20 con 0x7fb09c107a00 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.429+0000 7fb0a2e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c111e20 con 0x7fb09c102370 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.429+0000 7fb09be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c1019b0 0x7fb09c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:50.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09b400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb09c102370 0x7fb09c19b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c1019b0 0x7fb09c19b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53948/0 (socket says 172.21.15.16:53948) 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09be00700 1 -- 172.21.15.16:0/2798341795 learned_addr learned my addr 172.21.15.16:0/2798341795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb0a0e00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb09c107a00 0x7fb09c19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09b400700 1 -- 172.21.15.16:0/2798341795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb09c107a00 msgr2=0x7fb09c19fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09b400700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb09c107a00 0x7fb09c19fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09b400700 1 -- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c1019b0 msgr2=0x7fb09c19b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09b400700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c1019b0 0x7fb09c19b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09b400700 1 -- 172.21.15.16:0/2798341795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09c1a0430 con 0x7fb09c102370 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09be00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb09c1019b0 0x7fb09c19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.430+0000 7fb09b400700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb09c102370 0x7fb09c19b960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb08800e9f0 tx=0x7fb08800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.431+0000 7fb098c00700 1 -- 172.21.15.16:0/2798341795 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08800c9c0 con 0x7fb09c102370 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.431+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09c1a0720 con 0x7fb09c102370 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.431+0000 7fb098c00700 1 -- 172.21.15.16:0/2798341795 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb08800cb60 con 0x7fb09c102370 2024-04-13T03:32:50.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.431+0000 7fb098c00700 1 -- 172.21.15.16:0/2798341795 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb088018680 con 0x7fb09c102370 2024-04-13T03:32:50.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.431+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb09c1a7fa0 con 0x7fb09c102370 2024-04-13T03:32:50.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.432+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb09c06b1b0 con 0x7fb09c102370 2024-04-13T03:32:50.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.434+0000 7fb098c00700 1 -- 172.21.15.16:0/2798341795 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb088010430 con 0x7fb09c102370 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.436+0000 7fb098c00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0800a3cd0 0x7fb0800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.436+0000 7fb098c00700 1 -- 172.21.15.16:0/2798341795 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb0880d0a60 con 0x7fb09c102370 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.436+0000 7fb09be00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0800a3cd0 0x7fb0800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.436+0000 7fb098c00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] conn(0x7fb0800aee30 0x7fb0800b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.436+0000 7fb098c00700 1 -- 172.21.15.16:0/2798341795 --> [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb088015e80 con 0x7fb0800aee30 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.436+0000 7fb098c00700 1 -- 172.21.15.16:0/2798341795 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fb0880d52a0 con 0x7fb09c102370 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.436+0000 7fb0a0e00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] conn(0x7fb0800aee30 0x7fb0800b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.437+0000 7fb09be00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0800a3cd0 0x7fb0800a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb08c006010 tx=0x7fb08c009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.437+0000 7fb0a0e00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] conn(0x7fb0800aee30 0x7fb0800b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.438+0000 7fb098c00700 1 -- 172.21.15.16:0/2798341795 <== osd.0 v2:172.21.15.16:6802/4119728361 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb088015e80 con 0x7fb0800aee30 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.457+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 --> [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb09c1030d0 con 0x7fb0800aee30 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.457+0000 7fb098c00700 1 -- 172.21.15.16:0/2798341795 <== osd.0 v2:172.21.15.16:6802/4119728361 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb09c1030d0 con 0x7fb0800aee30 2024-04-13T03:32:50.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.458+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] conn(0x7fb0800aee30 msgr2=0x7fb0800b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.458+0000 7fb0a2e00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] conn(0x7fb0800aee30 0x7fb0800b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.458+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0800a3cd0 msgr2=0x7fb0800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.458+0000 7fb0a2e00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb0800a3cd0 0x7fb0800a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb08c006010 tx=0x7fb08c009570 comp rx=0 tx=0).stop 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.458+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb09c102370 msgr2=0x7fb09c19b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.458+0000 7fb0a2e00700 1 --2- 172.21.15.16:0/2798341795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb09c102370 0x7fb09c19b960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb08800e9f0 tx=0x7fb08800ed00 comp rx=0 tx=0).stop 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.458+0000 7fb0a0e00700 1 -- 172.21.15.16:0/2798341795 reap_dead start 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.459+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 shutdown_connections 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.459+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 >> 172.21.15.16:0/2798341795 conn(0x7fb09c078960 msgr2=0x7fb09c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.459+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 shutdown_connections 2024-04-13T03:32:50.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:50.459+0000 7fb0a2e00700 1 -- 172.21.15.16:0/2798341795 wait complete. 2024-04-13T03:32:50.831 INFO:teuthology.orchestra.run.smithi016.stdout:197568495671 2024-04-13T03:32:50.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-13T03:32:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:50 smithi016 ceph-mon[31772]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 6.8 KiB/s, 1 objects/s recovering 2024-04-13T03:32:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:50 smithi119 ceph-mon[35835]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 6.8 KiB/s, 1 objects/s recovering 2024-04-13T03:32:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:50 smithi188 ceph-mon[36398]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 6.8 KiB/s, 1 objects/s recovering 2024-04-13T03:32:51.766 INFO:teuthology.orchestra.run.smithi016.stdout:163208757309 2024-04-13T03:32:51.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-13T03:32:53.131 INFO:teuthology.orchestra.run.smithi016.stdout:313532612641 2024-04-13T03:32:53.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-13T03:32:53.224 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:52 smithi016 ceph-mon[31772]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 6.2 KiB/s, 1 objects/s recovering 2024-04-13T03:32:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:52 smithi119 ceph-mon[35835]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 6.2 KiB/s, 1 objects/s recovering 2024-04-13T03:32:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:52 smithi188 ceph-mon[36398]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 6.2 KiB/s, 1 objects/s recovering 2024-04-13T03:32:53.456 INFO:teuthology.orchestra.run.smithi016.stdout:141733920835 2024-04-13T03:32:53.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-13T03:32:55.083 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:54 smithi016 ceph-mon[31772]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s; 5.2 KiB/s, 0 objects/s recovering 2024-04-13T03:32:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:54 smithi119 ceph-mon[35835]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s; 5.2 KiB/s, 0 objects/s recovering 2024-04-13T03:32:55.348 INFO:teuthology.orchestra.run.smithi016.stdout:34359738458 2024-04-13T03:32:55.349 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-13T03:32:55.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:54 smithi188 ceph-mon[36398]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s; 5.2 KiB/s, 0 objects/s recovering 2024-04-13T03:32:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.036+0000 7f0c8ea00700 1 -- 172.21.15.16:0/2939576342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c8806b160 msgr2=0x7f0c8810f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.036+0000 7f0c8ea00700 1 --2- 172.21.15.16:0/2939576342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c8806b160 0x7f0c8810f970 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0c80009990 tx=0x7f0c80009ca0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.036+0000 7f0c8ea00700 1 -- 172.21.15.16:0/2939576342 shutdown_connections 2024-04-13T03:32:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.036+0000 7f0c8ea00700 1 --2- 172.21.15.16:0/2939576342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c8806b160 0x7f0c8810f970 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.036+0000 7f0c8ea00700 1 --2- 172.21.15.16:0/2939576342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c88108aa0 0x7f0c88108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.036+0000 7f0c8ea00700 1 --2- 172.21.15.16:0/2939576342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8806a7c0 0x7f0c8806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.036+0000 7f0c8ea00700 1 -- 172.21.15.16:0/2939576342 >> 172.21.15.16:0/2939576342 conn(0x7f0c88078960 msgr2=0x7f0c88078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.036+0000 7f0c8ea00700 1 -- 172.21.15.16:0/2939576342 shutdown_connections 2024-04-13T03:32:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.036+0000 7f0c8ea00700 1 -- 172.21.15.16:0/2939576342 wait complete. 2024-04-13T03:32:56.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.037+0000 7f0c8ea00700 1 Processor -- start 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.037+0000 7f0c8ea00700 1 -- start start 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c8ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c8806a7c0 0x7f0c88199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c8ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8806b160 0x7f0c88199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c8ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c88108aa0 0x7f0c8819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c8ea00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c88112dd0 con 0x7f0c88108aa0 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c8ea00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0c88112c50 con 0x7f0c8806a7c0 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c8ea00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0c88112f50 con 0x7f0c8806b160 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c87e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c8806a7c0 0x7f0c88199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c7f400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8806b160 0x7f0c88199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c8ca00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c88108aa0 0x7f0c8819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c87e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c8806a7c0 0x7f0c88199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:50890/0 (socket says 172.21.15.16:50890) 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c87e00700 1 -- 172.21.15.16:0/3703485736 learned_addr learned my addr 172.21.15.16:0/3703485736 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c87e00700 1 -- 172.21.15.16:0/3703485736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8806b160 msgr2=0x7f0c88199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c87e00700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8806b160 0x7f0c88199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c87e00700 1 -- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c88108aa0 msgr2=0x7f0c8819dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.038+0000 7f0c87e00700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c88108aa0 0x7f0c8819dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.039+0000 7f0c87e00700 1 -- 172.21.15.16:0/3703485736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8819e200 con 0x7f0c8806a7c0 2024-04-13T03:32:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.039+0000 7f0c7f400700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c8806b160 0x7f0c88199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:32:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.039+0000 7f0c87e00700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c8806a7c0 0x7f0c88199240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800ef10 tx=0x7f0c7800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:56.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.039+0000 7f0c85600700 1 -- 172.21.15.16:0/3703485736 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7800c9c0 con 0x7f0c8806a7c0 2024-04-13T03:32:56.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.040+0000 7f0c85600700 1 -- 172.21.15.16:0/3703485736 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c78019070 con 0x7f0c8806a7c0 2024-04-13T03:32:56.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.040+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c8819e490 con 0x7f0c8806a7c0 2024-04-13T03:32:56.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.040+0000 7f0c85600700 1 -- 172.21.15.16:0/3703485736 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c78007e60 con 0x7f0c8806a7c0 2024-04-13T03:32:56.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.040+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c881a5d70 con 0x7f0c8806a7c0 2024-04-13T03:32:56.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.040+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0c880631c0 con 0x7f0c8806a7c0 2024-04-13T03:32:56.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.044+0000 7f0c85600700 1 -- 172.21.15.16:0/3703485736 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0c78012050 con 0x7f0c8806a7c0 2024-04-13T03:32:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.045+0000 7f0c85600700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0c6c0a3d50 0x7f0c6c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.046+0000 7f0c85600700 1 -- 172.21.15.16:0/3703485736 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f0c780d1260 con 0x7f0c8806a7c0 2024-04-13T03:32:56.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.046+0000 7f0c7f400700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0c6c0a3d50 0x7f0c6c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.046+0000 7f0c85600700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] conn(0x7f0c6c0aeeb0 0x7f0c6c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.046+0000 7f0c85600700 1 -- 172.21.15.16:0/3703485736 --> [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c6c0b1a20 con 0x7f0c6c0aeeb0 2024-04-13T03:32:56.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.046+0000 7f0c85600700 1 -- 172.21.15.16:0/3703485736 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f0c780d1610 con 0x7f0c8806a7c0 2024-04-13T03:32:56.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.046+0000 7f0c8ca00700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] conn(0x7f0c6c0aeeb0 0x7f0c6c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.047+0000 7f0c7f400700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0c6c0a3d50 0x7f0c6c0a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0c88078ef0 tx=0x7f0c70009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:56.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.047+0000 7f0c8ca00700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] conn(0x7f0c6c0aeeb0 0x7f0c6c0b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:56.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.048+0000 7f0c85600700 1 -- 172.21.15.16:0/3703485736 <== osd.9 v2:172.21.15.16:6826/3476141905 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c6c0b1a20 con 0x7f0c6c0aeeb0 2024-04-13T03:32:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.066+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 --> [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0c880635c0 con 0x7f0c6c0aeeb0 2024-04-13T03:32:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.066+0000 7f0c85600700 1 -- 172.21.15.16:0/3703485736 <== osd.9 v2:172.21.15.16:6826/3476141905 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0c880635c0 con 0x7f0c6c0aeeb0 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.067+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] conn(0x7f0c6c0aeeb0 msgr2=0x7f0c6c0b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.067+0000 7f0c8ea00700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] conn(0x7f0c6c0aeeb0 0x7f0c6c0b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.067+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0c6c0a3d50 msgr2=0x7f0c6c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.067+0000 7f0c8ea00700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0c6c0a3d50 0x7f0c6c0a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0c88078ef0 tx=0x7f0c70009260 comp rx=0 tx=0).stop 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.067+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c8806a7c0 msgr2=0x7f0c88199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.067+0000 7f0c8ea00700 1 --2- 172.21.15.16:0/3703485736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c8806a7c0 0x7f0c88199240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800ef10 tx=0x7f0c7800c3a0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.067+0000 7f0c8ca00700 1 -- 172.21.15.16:0/3703485736 reap_dead start 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.067+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 shutdown_connections 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.067+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 >> 172.21.15.16:0/3703485736 conn(0x7f0c88078960 msgr2=0x7f0c880ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.068+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 shutdown_connections 2024-04-13T03:32:56.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:56.068+0000 7f0c8ea00700 1 -- 172.21.15.16:0/3703485736 wait complete. 2024-04-13T03:32:56.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.748+0000 7f5642200700 1 -- 172.21.15.16:0/2333868202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f563c103510 msgr2=0x7f563c10f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.748+0000 7f5642200700 1 --2- 172.21.15.16:0/2333868202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f563c103510 0x7f563c10f9b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f563400b0a0 tx=0x7f563400b3b0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.748+0000 7f5642200700 1 -- 172.21.15.16:0/2333868202 shutdown_connections 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.748+0000 7f5642200700 1 --2- 172.21.15.16:0/2333868202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f563c103510 0x7f563c10f9b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.748+0000 7f5642200700 1 --2- 172.21.15.16:0/2333868202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f563c108be0 0x7f563c108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.748+0000 7f5642200700 1 --2- 172.21.15.16:0/2333868202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c102b50 0x7f563c102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.748+0000 7f5642200700 1 -- 172.21.15.16:0/2333868202 >> 172.21.15.16:0/2333868202 conn(0x7f563c079c60 msgr2=0x7f563c07a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.748+0000 7f5642200700 1 -- 172.21.15.16:0/2333868202 shutdown_connections 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.749+0000 7f5642200700 1 -- 172.21.15.16:0/2333868202 wait complete. 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.750+0000 7f5642200700 1 Processor -- start 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.750+0000 7f5642200700 1 -- start start 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f5642200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c102b50 0x7f563c19b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f5642200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f563c103510 0x7f563c19ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f5642200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f563c108be0 0x7f563c19fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f5642200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f563c112e50 con 0x7f563c108be0 2024-04-13T03:32:56.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f5642200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f563c112cd0 con 0x7f563c102b50 2024-04-13T03:32:56.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f5642200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f563c112fd0 con 0x7f563c103510 2024-04-13T03:32:56.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f563aa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f563c103510 0x7f563c19ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f563b400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c102b50 0x7f563c19b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f563be00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f563c108be0 0x7f563c19fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f563aa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f563c103510 0x7f563c19ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:38146/0 (socket says 172.21.15.16:38146) 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f563b400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c102b50 0x7f563c19b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:50880/0 (socket says 172.21.15.16:50880) 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.751+0000 7f563aa00700 1 -- 172.21.15.16:0/1001021429 learned_addr learned my addr 172.21.15.16:0/1001021429 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f563aa00700 1 -- 172.21.15.16:0/1001021429 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c102b50 msgr2=0x7f563c19b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f563aa00700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f563c102b50 0x7f563c19b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f563aa00700 1 -- 172.21.15.16:0/1001021429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f563c108be0 msgr2=0x7f563c19fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f563aa00700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f563c108be0 0x7f563c19fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f563aa00700 1 -- 172.21.15.16:0/1001021429 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563c1a0510 con 0x7f563c103510 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f563aa00700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f563c103510 0x7f563c19ba40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f562800e9c0 tx=0x7f562800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f5633e00700 1 -- 172.21.15.16:0/1001021429 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562800cbd0 con 0x7f563c103510 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f5633e00700 1 -- 172.21.15.16:0/1001021429 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562800ee60 con 0x7f563c103510 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f5633e00700 1 -- 172.21.15.16:0/1001021429 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628005630 con 0x7f563c103510 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.752+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563c1a0800 con 0x7f563c103510 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.753+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f563c1a8080 con 0x7f563c103510 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.754+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f563c10f920 con 0x7f563c103510 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.756+0000 7f5633e00700 1 -- 172.21.15.16:0/1001021429 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5628004750 con 0x7f563c103510 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.758+0000 7f5633e00700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f56200a3d50 0x7f56200a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.758+0000 7f5633e00700 1 -- 172.21.15.16:0/1001021429 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f56280d0e40 con 0x7f563c103510 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.758+0000 7f563b400700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f56200a3d50 0x7f56200a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.758+0000 7f5633e00700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] conn(0x7f56200aeeb0 0x7f56200b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.758+0000 7f5633e00700 1 -- 172.21.15.16:0/1001021429 --> [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f562801ee50 con 0x7f56200aeeb0 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.758+0000 7f5633e00700 1 -- 172.21.15.16:0/1001021429 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f56280d11f0 con 0x7f563c103510 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.759+0000 7f563be00700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] conn(0x7f56200aeeb0 0x7f56200b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.759+0000 7f563b400700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f56200a3d50 0x7f56200a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f562c007b60 tx=0x7f562c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:56.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.759+0000 7f563be00700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] conn(0x7f56200aeeb0 0x7f56200b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.760+0000 7f5633e00700 1 -- 172.21.15.16:0/1001021429 <== osd.1 v2:172.21.15.119:6800/1083610996 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f562801ee50 con 0x7f56200aeeb0 2024-04-13T03:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.780+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 --> [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f563c1041e0 con 0x7f56200aeeb0 2024-04-13T03:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.781+0000 7f5633e00700 1 -- 172.21.15.16:0/1001021429 <== osd.1 v2:172.21.15.119:6800/1083610996 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f563c1041e0 con 0x7f56200aeeb0 2024-04-13T03:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.781+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 >> [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] conn(0x7f56200aeeb0 msgr2=0x7f56200b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.781+0000 7f5642200700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] conn(0x7f56200aeeb0 0x7f56200b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.782+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f56200a3d50 msgr2=0x7f56200a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.782+0000 7f5642200700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f56200a3d50 0x7f56200a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f562c007b60 tx=0x7f562c0073d0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.782+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f563c103510 msgr2=0x7f563c19ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.782+0000 7f5642200700 1 --2- 172.21.15.16:0/1001021429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f563c103510 0x7f563c19ba40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f562800e9c0 tx=0x7f562800ecd0 comp rx=0 tx=0).stop 2024-04-13T03:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.782+0000 7f563be00700 1 -- 172.21.15.16:0/1001021429 reap_dead start 2024-04-13T03:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.782+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 shutdown_connections 2024-04-13T03:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.782+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 >> 172.21.15.16:0/1001021429 conn(0x7f563c079c60 msgr2=0x7f563c10df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.782+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 shutdown_connections 2024-04-13T03:32:56.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:32:54.782+0000 7f5642200700 1 -- 172.21.15.16:0/1001021429 wait complete. 2024-04-13T03:32:57.239 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:56 smithi016 ceph-mon[31772]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s; 5.2 KiB/s, 0 objects/s recovering 2024-04-13T03:32:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:56 smithi119 ceph-mon[35835]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s; 5.2 KiB/s, 0 objects/s recovering 2024-04-13T03:32:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:56 smithi188 ceph-mon[36398]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s; 5.2 KiB/s, 0 objects/s recovering 2024-04-13T03:32:57.691 INFO:teuthology.orchestra.run.smithi016.stdout:55834574934 2024-04-13T03:32:57.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-13T03:32:58.066 INFO:teuthology.orchestra.run.smithi016.stdout:287762808873 2024-04-13T03:32:58.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-13T03:32:58.098 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:58.099 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:59.077 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:59.079 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:32:59.311 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:32:58 smithi016 ceph-mon[31772]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:32:59.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:32:58 smithi119 ceph-mon[35835]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:32:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:32:58 smithi188 ceph-mon[36398]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:01.271 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:00 smithi016 ceph-mon[31772]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-13T03:33:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:00 smithi119 ceph-mon[35835]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-13T03:33:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:00 smithi188 ceph-mon[36398]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-13T03:33:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.274+0000 7f6e0fc00700 1 -- 172.21.15.16:0/3391272188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075600 msgr2=0x7f6e0810ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.274+0000 7f6e0fc00700 1 --2- 172.21.15.16:0/3391272188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075600 0x7f6e0810ef70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009990 tx=0x7f6df4009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.274+0000 7f6e0fc00700 1 -- 172.21.15.16:0/3391272188 shutdown_connections 2024-04-13T03:33:02.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.274+0000 7f6e0fc00700 1 --2- 172.21.15.16:0/3391272188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e0810f4b0 0x7f6e081118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.275+0000 7f6e0fc00700 1 --2- 172.21.15.16:0/3391272188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e08074ce0 0x7f6e080750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.275+0000 7f6e0fc00700 1 --2- 172.21.15.16:0/3391272188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075600 0x7f6e0810ef70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.275+0000 7f6e0fc00700 1 -- 172.21.15.16:0/3391272188 >> 172.21.15.16:0/3391272188 conn(0x7f6e08079c10 msgr2=0x7f6e0807a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.275+0000 7f6e0fc00700 1 -- 172.21.15.16:0/3391272188 shutdown_connections 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.275+0000 7f6e0fc00700 1 -- 172.21.15.16:0/3391272188 wait complete. 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.276+0000 7f6e0fc00700 1 Processor -- start 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.276+0000 7f6e0fc00700 1 -- start start 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0fc00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08074ce0 0x7f6e0819b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0fc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e08075600 0x7f6e0819ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0fc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e0810f4b0 0x7f6e0819fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0fc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08115020 con 0x7f6e08075600 2024-04-13T03:33:02.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0fc00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08114ea0 con 0x7f6e08074ce0 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0fc00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081151a0 con 0x7f6e0810f4b0 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0dc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e0810f4b0 0x7f6e0819fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0dc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e0810f4b0 0x7f6e0819fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:51946/0 (socket says 172.21.15.16:51946) 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0d200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08074ce0 0x7f6e0819b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.277+0000 7f6e0dc00700 1 -- 172.21.15.16:0/1998696241 learned_addr learned my addr 172.21.15.16:0/1998696241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.278+0000 7f6e07e00700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e08075600 0x7f6e0819ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.278+0000 7f6e0d200700 1 -- 172.21.15.16:0/1998696241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e0810f4b0 msgr2=0x7f6e0819fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.278+0000 7f6e0d200700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e0810f4b0 0x7f6e0819fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.278+0000 7f6e0d200700 1 -- 172.21.15.16:0/1998696241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e08075600 msgr2=0x7f6e0819ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.278+0000 7f6e0d200700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e08075600 0x7f6e0819ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.278+0000 7f6e0d200700 1 -- 172.21.15.16:0/1998696241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e081a04e0 con 0x7f6e08074ce0 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.278+0000 7f6e07e00700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e08075600 0x7f6e0819ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-13T03:33:02.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.278+0000 7f6e0d200700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08074ce0 0x7f6e0819b4d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00e9f0 tx=0x7f6dfc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:02.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.278+0000 7f6e05600700 1 -- 172.21.15.16:0/1998696241 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc00cbd0 con 0x7f6e08074ce0 2024-04-13T03:33:02.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.279+0000 7f6e05600700 1 -- 172.21.15.16:0/1998696241 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dfc004500 con 0x7f6e08074ce0 2024-04-13T03:33:02.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.279+0000 7f6e05600700 1 -- 172.21.15.16:0/1998696241 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc005280 con 0x7f6e08074ce0 2024-04-13T03:33:02.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.279+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e081a07d0 con 0x7f6e08074ce0 2024-04-13T03:33:02.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.279+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e081a8050 con 0x7f6e08074ce0 2024-04-13T03:33:02.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.280+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6e08063240 con 0x7f6e08074ce0 2024-04-13T03:33:02.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.282+0000 7f6e05600700 1 -- 172.21.15.16:0/1998696241 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6dfc005420 con 0x7f6e08074ce0 2024-04-13T03:33:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.283+0000 7f6e05600700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6df00a3d50 0x7f6df00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.284+0000 7f6e05600700 1 -- 172.21.15.16:0/1998696241 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f6dfc0d0b10 con 0x7f6e08074ce0 2024-04-13T03:33:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.284+0000 7f6e07e00700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6df00a3d50 0x7f6df00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.284+0000 7f6e05600700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] conn(0x7f6df00aeeb0 0x7f6df00b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.284+0000 7f6e05600700 1 -- 172.21.15.16:0/1998696241 --> [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6dfc00be50 con 0x7f6df00aeeb0 2024-04-13T03:33:02.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.284+0000 7f6e05600700 1 -- 172.21.15.16:0/1998696241 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f6dfc0d0ec0 con 0x7f6e08074ce0 2024-04-13T03:33:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.285+0000 7f6e0dc00700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] conn(0x7f6df00aeeb0 0x7f6df00b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.285+0000 7f6e07e00700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6df00a3d50 0x7f6df00a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6df400b2e0 tx=0x7f6df4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:02.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.285+0000 7f6e0dc00700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] conn(0x7f6df00aeeb0 0x7f6df00b1350 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:02.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.286+0000 7f6e05600700 1 -- 172.21.15.16:0/1998696241 <== osd.8 v2:172.21.15.188:6816/3338270245 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6dfc00be50 con 0x7f6df00aeeb0 2024-04-13T03:33:02.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.304+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 --> [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6e080636c0 con 0x7f6df00aeeb0 2024-04-13T03:33:02.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.305+0000 7f6e05600700 1 -- 172.21.15.16:0/1998696241 <== osd.8 v2:172.21.15.188:6816/3338270245 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6e080636c0 con 0x7f6df00aeeb0 2024-04-13T03:33:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 >> [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] conn(0x7f6df00aeeb0 msgr2=0x7f6df00b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] conn(0x7f6df00aeeb0 0x7f6df00b1350 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6df00a3d50 msgr2=0x7f6df00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:02.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6df00a3d50 0x7f6df00a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6df400b2e0 tx=0x7f6df4005fb0 comp rx=0 tx=0).stop 2024-04-13T03:33:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08074ce0 msgr2=0x7f6e0819b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 --2- 172.21.15.16:0/1998696241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08074ce0 0x7f6e0819b4d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00e9f0 tx=0x7f6dfc00ed00 comp rx=0 tx=0).stop 2024-04-13T03:33:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0dc00700 1 -- 172.21.15.16:0/1998696241 reap_dead start 2024-04-13T03:33:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 shutdown_connections 2024-04-13T03:33:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 >> 172.21.15.16:0/1998696241 conn(0x7f6e08079c10 msgr2=0x7f6e080ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 shutdown_connections 2024-04-13T03:33:02.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:02.306+0000 7f6e0fc00700 1 -- 172.21.15.16:0/1998696241 wait complete. 2024-04-13T03:33:02.466 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:03.253 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:02 smithi016 ceph-mon[31772]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:02 smithi119 ceph-mon[35835]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:02 smithi188 ceph-mon[36398]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:05.092 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:04 smithi016 ceph-mon[31772]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:05 smithi119 ceph-mon[35835]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:05 smithi188 ceph-mon[36398]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:07 smithi119 ceph-mon[35835]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:33:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:07 smithi016 ceph-mon[31772]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:33:07.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:07 smithi188 ceph-mon[36398]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:33:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:09 smithi119 ceph-mon[35835]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:09.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:09 smithi016 ceph-mon[31772]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:09 smithi188 ceph-mon[36398]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:10.273 INFO:teuthology.orchestra.run.smithi016.stdout:253403070513 2024-04-13T03:33:10.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-13T03:33:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.662+0000 7f4225000700 1 -- 172.21.15.16:0/2836826305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f422006a7c0 msgr2=0x7f422006aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.662+0000 7f4225000700 1 --2- 172.21.15.16:0/2836826305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f422006a7c0 0x7f422006aba0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f42100098f0 tx=0x7f4210009c00 comp rx=0 tx=0).stop 2024-04-13T03:33:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.662+0000 7f4225000700 1 -- 172.21.15.16:0/2836826305 shutdown_connections 2024-04-13T03:33:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.662+0000 7f4225000700 1 --2- 172.21.15.16:0/2836826305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422010f560 0x7f4220111950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.662+0000 7f4225000700 1 --2- 172.21.15.16:0/2836826305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f422006a7c0 0x7f422006aba0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.662+0000 7f4225000700 1 --2- 172.21.15.16:0/2836826305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f422006b0e0 0x7f422010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.662+0000 7f4225000700 1 -- 172.21.15.16:0/2836826305 >> 172.21.15.16:0/2836826305 conn(0x7f4220078960 msgr2=0x7f4220078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.662+0000 7f4225000700 1 -- 172.21.15.16:0/2836826305 shutdown_connections 2024-04-13T03:33:10.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.662+0000 7f4225000700 1 -- 172.21.15.16:0/2836826305 wait complete. 2024-04-13T03:33:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.663+0000 7f4225000700 1 Processor -- start 2024-04-13T03:33:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f4225000700 1 -- start start 2024-04-13T03:33:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f4225000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f422006a7c0 0x7f422019b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f4225000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422006b0e0 0x7f422019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f4225000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f422010f560 0x7f422019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f4225000700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4220114fe0 con 0x7f422006b0e0 2024-04-13T03:33:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f4225000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4220114e60 con 0x7f422010f560 2024-04-13T03:33:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f4225000700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4220115160 con 0x7f422006a7c0 2024-04-13T03:33:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f421d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422006b0e0 0x7f422019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f421d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422006b0e0 0x7f422019b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55002/0 (socket says 172.21.15.16:55002) 2024-04-13T03:33:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f421de00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f422006a7c0 0x7f422019b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.664+0000 7f421d400700 1 -- 172.21.15.16:0/3310374427 learned_addr learned my addr 172.21.15.16:0/3310374427 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.665+0000 7f421e800700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f422010f560 0x7f422019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.665+0000 7f421de00700 1 -- 172.21.15.16:0/3310374427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f422010f560 msgr2=0x7f422019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.665+0000 7f421de00700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f422010f560 0x7f422019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.665+0000 7f421de00700 1 -- 172.21.15.16:0/3310374427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422006b0e0 msgr2=0x7f422019b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.665+0000 7f421de00700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422006b0e0 0x7f422019b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.665+0000 7f421de00700 1 -- 172.21.15.16:0/3310374427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42201a04c0 con 0x7f422006a7c0 2024-04-13T03:33:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.665+0000 7f421de00700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f422006a7c0 0x7f422019b4b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f421000c010 tx=0x7f421000fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.666+0000 7f4216a00700 1 -- 172.21.15.16:0/3310374427 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4210005180 con 0x7f422006a7c0 2024-04-13T03:33:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.666+0000 7f4216a00700 1 -- 172.21.15.16:0/3310374427 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f421000b560 con 0x7f422006a7c0 2024-04-13T03:33:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.666+0000 7f4216a00700 1 -- 172.21.15.16:0/3310374427 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4210003a80 con 0x7f422006a7c0 2024-04-13T03:33:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.666+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42201a0750 con 0x7f422006a7c0 2024-04-13T03:33:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.666+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42201a8030 con 0x7f422006a7c0 2024-04-13T03:33:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.667+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42200697a0 con 0x7f422006a7c0 2024-04-13T03:33:10.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.669+0000 7f4216a00700 1 -- 172.21.15.16:0/3310374427 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f421000b710 con 0x7f422006a7c0 2024-04-13T03:33:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.671+0000 7f4216a00700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f42040a3cd0 0x7f42040a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.671+0000 7f4216a00700 1 -- 172.21.15.16:0/3310374427 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f42100d1b80 con 0x7f422006a7c0 2024-04-13T03:33:10.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.671+0000 7f421d400700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f42040a3cd0 0x7f42040a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:10.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.672+0000 7f421d400700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f42040a3cd0 0x7f42040a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4220100b60 tx=0x7f420c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:10.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.673+0000 7f4216a00700 1 -- 172.21.15.16:0/3310374427 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4210013030 con 0x7f422006a7c0 2024-04-13T03:33:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.908+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f422002dfa0 con 0x7f422006a7c0 2024-04-13T03:33:10.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.909+0000 7f4216a00700 1 -- 172.21.15.16:0/3310374427 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f421009e0d0 con 0x7f422006a7c0 2024-04-13T03:33:10.910 INFO:teuthology.orchestra.run.smithi016.stdout:347892351009 2024-04-13T03:33:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f42040a3cd0 msgr2=0x7f42040a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f42040a3cd0 0x7f42040a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4220100b60 tx=0x7f420c009260 comp rx=0 tx=0).stop 2024-04-13T03:33:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f422006a7c0 msgr2=0x7f422019b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:10.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f422006a7c0 0x7f422019b4b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f421000c010 tx=0x7f421000fa70 comp rx=0 tx=0).stop 2024-04-13T03:33:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 shutdown_connections 2024-04-13T03:33:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f42040a3cd0 0x7f42040a6190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f422010f560 0x7f422019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f422006a7c0 0x7f422019b4b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 --2- 172.21.15.16:0/3310374427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f422006b0e0 0x7f422019b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 >> 172.21.15.16:0/3310374427 conn(0x7f4220078960 msgr2=0x7f4220110a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.912+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 shutdown_connections 2024-04-13T03:33:10.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:10.913+0000 7f4225000700 1 -- 172.21.15.16:0/3310374427 wait complete. 2024-04-13T03:33:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:11 smithi119 ceph-mon[35835]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:33:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:11 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3310374427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-13T03:33:11.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:11 smithi016 ceph-mon[31772]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:33:11.347 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:11 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3310374427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-13T03:33:11.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:11 smithi188 ceph-mon[36398]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:33:11.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:11 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3310374427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-13T03:33:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.522+0000 7f1e19c00700 1 -- 172.21.15.16:0/3435858860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e14109340 msgr2=0x7f1e14109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.522+0000 7f1e19c00700 1 --2- 172.21.15.16:0/3435858860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e14109340 0x7f1e14109720 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e04009990 tx=0x7f1e04009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.522+0000 7f1e19c00700 1 -- 172.21.15.16:0/3435858860 shutdown_connections 2024-04-13T03:33:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.522+0000 7f1e19c00700 1 --2- 172.21.15.16:0/3435858860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e1410e420 0x7f1e14115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.522+0000 7f1e19c00700 1 --2- 172.21.15.16:0/3435858860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e14109340 0x7f1e14109720 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.522+0000 7f1e19c00700 1 --2- 172.21.15.16:0/3435858860 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14109cf0 0x7f1e1410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.522+0000 7f1e19c00700 1 -- 172.21.15.16:0/3435858860 >> 172.21.15.16:0/3435858860 conn(0x7f1e14076280 msgr2=0x7f1e140786a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.522+0000 7f1e19c00700 1 -- 172.21.15.16:0/3435858860 shutdown_connections 2024-04-13T03:33:11.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.522+0000 7f1e19c00700 1 -- 172.21.15.16:0/3435858860 wait complete. 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.523+0000 7f1e19c00700 1 Processor -- start 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e19c00700 1 -- start start 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e19c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e14109340 0x7f1e141a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e19c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14109cf0 0x7f1e141a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e19c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e1410e420 0x7f1e1419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e19c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e14119480 con 0x7f1e14109cf0 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e19c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1e14119300 con 0x7f1e1410e420 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e19c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1e14119600 con 0x7f1e14109340 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e12000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14109cf0 0x7f1e141a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e12000700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14109cf0 0x7f1e141a5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55026/0 (socket says 172.21.15.16:55026) 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e12000700 1 -- 172.21.15.16:0/1894331804 learned_addr learned my addr 172.21.15.16:0/1894331804 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.524+0000 7f1e12a00700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e14109340 0x7f1e141a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.525+0000 7f1e13400700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e1410e420 0x7f1e1419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.525+0000 7f1e12000700 1 -- 172.21.15.16:0/1894331804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e14109340 msgr2=0x7f1e141a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.525+0000 7f1e12000700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e14109340 0x7f1e141a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.525+0000 7f1e12000700 1 -- 172.21.15.16:0/1894331804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e1410e420 msgr2=0x7f1e1419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.525+0000 7f1e12000700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e1410e420 0x7f1e1419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.525+0000 7f1e12000700 1 -- 172.21.15.16:0/1894331804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1419ff80 con 0x7f1e14109cf0 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.525+0000 7f1e12000700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14109cf0 0x7f1e141a5b60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000e970 tx=0x7f1e0000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.526+0000 7f1e0b400700 1 -- 172.21.15.16:0/1894331804 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00004a30 con 0x7f1e14109cf0 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.526+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e141a0270 con 0x7f1e14109cf0 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.526+0000 7f1e0b400700 1 -- 172.21.15.16:0/1894331804 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e0000ccc0 con 0x7f1e14109cf0 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.526+0000 7f1e0b400700 1 -- 172.21.15.16:0/1894331804 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00005650 con 0x7f1e14109cf0 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.526+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e141ac480 con 0x7f1e14109cf0 2024-04-13T03:33:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.527+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e1410a820 con 0x7f1e14109cf0 2024-04-13T03:33:11.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.530+0000 7f1e0b400700 1 -- 172.21.15.16:0/1894331804 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1e00004bd0 con 0x7f1e14109cf0 2024-04-13T03:33:11.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.531+0000 7f1e0b400700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1df80a3c40 0x7f1df80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:11.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.531+0000 7f1e0b400700 1 -- 172.21.15.16:0/1894331804 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f1e000d09a0 con 0x7f1e14109cf0 2024-04-13T03:33:11.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.532+0000 7f1e12a00700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1df80a3c40 0x7f1df80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:11.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.532+0000 7f1e12a00700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1df80a3c40 0x7f1df80a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400b2e0 tx=0x7f1e04005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.536+0000 7f1e0b400700 1 -- 172.21.15.16:0/1894331804 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e0009d020 con 0x7f1e14109cf0 2024-04-13T03:33:11.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.782+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1e1402dfa0 con 0x7f1e14109cf0 2024-04-13T03:33:11.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.783+0000 7f1e0b400700 1 -- 172.21.15.16:0/1894331804 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1e0009c920 con 0x7f1e14109cf0 2024-04-13T03:33:11.784 INFO:teuthology.orchestra.run.smithi016.stdout:219043332152 2024-04-13T03:33:11.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.785+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1df80a3c40 msgr2=0x7f1df80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:11.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.785+0000 7f1e19c00700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1df80a3c40 0x7f1df80a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400b2e0 tx=0x7f1e04005fd0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.785+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14109cf0 msgr2=0x7f1e141a5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:11.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.785+0000 7f1e19c00700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14109cf0 0x7f1e141a5b60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000e970 tx=0x7f1e0000ec80 comp rx=0 tx=0).stop 2024-04-13T03:33:11.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.786+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 shutdown_connections 2024-04-13T03:33:11.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.786+0000 7f1e19c00700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1df80a3c40 0x7f1df80a6100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.786+0000 7f1e19c00700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e1410e420 0x7f1e1419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.786+0000 7f1e19c00700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e14109340 0x7f1e141a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.786+0000 7f1e19c00700 1 --2- 172.21.15.16:0/1894331804 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e14109cf0 0x7f1e141a5b60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:11.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.786+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 >> 172.21.15.16:0/1894331804 conn(0x7f1e14076280 msgr2=0x7f1e14078070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:11.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.786+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 shutdown_connections 2024-04-13T03:33:11.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:11.786+0000 7f1e19c00700 1 -- 172.21.15.16:0/1894331804 wait complete. 2024-04-13T03:33:12.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:12 smithi016 ceph-mon[31772]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-13T03:33:12.132 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:12 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1894331804' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-13T03:33:12.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:12 smithi119 ceph-mon[35835]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-13T03:33:12.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:12 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1894331804' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-13T03:33:12.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:12 smithi188 ceph-mon[36398]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-13T03:33:12.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:12 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1894331804' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-13T03:33:12.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.705+0000 7efd2a200700 1 -- 172.21.15.16:0/2122003463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd24108ac0 msgr2=0x7efd24108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:12.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.705+0000 7efd2a200700 1 --2- 172.21.15.16:0/2122003463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd24108ac0 0x7efd24108ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efd14009990 tx=0x7efd14009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.705+0000 7efd2a200700 1 -- 172.21.15.16:0/2122003463 shutdown_connections 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.705+0000 7efd2a200700 1 --2- 172.21.15.16:0/2122003463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd241034c0 0x7efd2410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.705+0000 7efd2a200700 1 --2- 172.21.15.16:0/2122003463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd24108ac0 0x7efd24108ea0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.705+0000 7efd2a200700 1 --2- 172.21.15.16:0/2122003463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd24102b00 0x7efd24102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.705+0000 7efd2a200700 1 -- 172.21.15.16:0/2122003463 >> 172.21.15.16:0/2122003463 conn(0x7efd24079c10 msgr2=0x7efd2407a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.705+0000 7efd2a200700 1 -- 172.21.15.16:0/2122003463 shutdown_connections 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.705+0000 7efd2a200700 1 -- 172.21.15.16:0/2122003463 wait complete. 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.706+0000 7efd2a200700 1 Processor -- start 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.706+0000 7efd2a200700 1 -- start start 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd2a200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd24102b00 0x7efd24199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd2a200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd241034c0 0x7efd24199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd2a200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd24108ac0 0x7efd2419dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd2a200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efd24112df0 con 0x7efd24108ac0 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd2a200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efd24112c70 con 0x7efd241034c0 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd2a200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7efd24112f70 con 0x7efd24102b00 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd23400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd24102b00 0x7efd24199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd23400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd24102b00 0x7efd24199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:39128/0 (socket says 172.21.15.16:39128) 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd23e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd24108ac0 0x7efd2419dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd23400700 1 -- 172.21.15.16:0/101861007 learned_addr learned my addr 172.21.15.16:0/101861007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.707+0000 7efd22a00700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd241034c0 0x7efd24199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.708+0000 7efd23400700 1 -- 172.21.15.16:0/101861007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd241034c0 msgr2=0x7efd24199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.708+0000 7efd23400700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd241034c0 0x7efd24199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.708+0000 7efd23400700 1 -- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd24108ac0 msgr2=0x7efd2419dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.708+0000 7efd23400700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd24108ac0 0x7efd2419dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.708+0000 7efd23400700 1 -- 172.21.15.16:0/101861007 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2419e330 con 0x7efd24102b00 2024-04-13T03:33:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.708+0000 7efd23e00700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd24108ac0 0x7efd2419dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.708+0000 7efd23400700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd24102b00 0x7efd24199320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efd14000c00 tx=0x7efd1400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.708+0000 7efd1be00700 1 -- 172.21.15.16:0/101861007 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd14027070 con 0x7efd24102b00 2024-04-13T03:33:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.709+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2419e5c0 con 0x7efd24102b00 2024-04-13T03:33:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.709+0000 7efd1be00700 1 -- 172.21.15.16:0/101861007 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd140056d0 con 0x7efd24102b00 2024-04-13T03:33:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.709+0000 7efd1be00700 1 -- 172.21.15.16:0/101861007 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd14022e10 con 0x7efd24102b00 2024-04-13T03:33:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.709+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd241a5ea0 con 0x7efd24102b00 2024-04-13T03:33:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.710+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd241041f0 con 0x7efd24102b00 2024-04-13T03:33:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.714+0000 7efd1be00700 1 -- 172.21.15.16:0/101861007 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efd1402a070 con 0x7efd24102b00 2024-04-13T03:33:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.715+0000 7efd1be00700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efd080a3c40 0x7efd080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.716+0000 7efd1be00700 1 -- 172.21.15.16:0/101861007 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7efd140d19d0 con 0x7efd24102b00 2024-04-13T03:33:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.716+0000 7efd22a00700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efd080a3c40 0x7efd080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.716+0000 7efd1be00700 1 -- 172.21.15.16:0/101861007 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd14013070 con 0x7efd24102b00 2024-04-13T03:33:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.717+0000 7efd22a00700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efd080a3c40 0x7efd080a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efd2407a2c0 tx=0x7efd10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.963+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7efd2402dfa0 con 0x7efd24102b00 2024-04-13T03:33:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.964+0000 7efd1be00700 1 -- 172.21.15.16:0/101861007 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7efd1409e0e0 con 0x7efd24102b00 2024-04-13T03:33:12.965 INFO:teuthology.orchestra.run.smithi016.stdout:197568495677 2024-04-13T03:33:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.967+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efd080a3c40 msgr2=0x7efd080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.967+0000 7efd2a200700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efd080a3c40 0x7efd080a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efd2407a2c0 tx=0x7efd10009260 comp rx=0 tx=0).stop 2024-04-13T03:33:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.967+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd24102b00 msgr2=0x7efd24199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.967+0000 7efd2a200700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd24102b00 0x7efd24199320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efd14000c00 tx=0x7efd1400bab0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.967+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 shutdown_connections 2024-04-13T03:33:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.968+0000 7efd2a200700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7efd080a3c40 0x7efd080a6100 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.968+0000 7efd2a200700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd24108ac0 0x7efd2419dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.968+0000 7efd2a200700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd24102b00 0x7efd24199320 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.968+0000 7efd2a200700 1 --2- 172.21.15.16:0/101861007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd241034c0 0x7efd24199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.968+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 >> 172.21.15.16:0/101861007 conn(0x7efd24079c10 msgr2=0x7efd24101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.968+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 shutdown_connections 2024-04-13T03:33:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.968+0000 7efd2a200700 1 -- 172.21.15.16:0/101861007 wait complete. 2024-04-13T03:33:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:13 smithi119 ceph-mon[35835]: pgmap v368: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-13T03:33:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:13 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/101861007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-13T03:33:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:13 smithi016 ceph-mon[31772]: pgmap v368: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-13T03:33:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:13 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/101861007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-13T03:33:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:13 smithi188 ceph-mon[36398]: pgmap v368: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-13T03:33:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:13 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/101861007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-13T03:33:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.839+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3367106291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eec108b80 msgr2=0x7f1eec108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.839+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3367106291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eec108b80 0x7f1eec108f60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0098f0 tx=0x7f1edc009c00 comp rx=0 tx=0).stop 2024-04-13T03:33:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.839+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3367106291 shutdown_connections 2024-04-13T03:33:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.839+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3367106291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1eec1034c0 0x7f1eec10f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:13.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.839+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3367106291 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eec108b80 0x7f1eec108f60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.839+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3367106291 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec102b00 0x7f1eec102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.839+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3367106291 >> 172.21.15.16:0/3367106291 conn(0x7f1eec079c60 msgr2=0x7f1eec07a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.839+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3367106291 shutdown_connections 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.839+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3367106291 wait complete. 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.841+0000 7f1ef0e00700 1 Processor -- start 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.841+0000 7f1ef0e00700 1 -- start start 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.841+0000 7f1ef0e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eec102b00 0x7f1eec074db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.841+0000 7f1ef0e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec1034c0 0x7f1eec0752f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.841+0000 7f1ef0e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1eec108b80 0x7f1eec075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.841+0000 7f1ef0e00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec112e30 con 0x7f1eec1034c0 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.841+0000 7f1ef0e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec112cb0 con 0x7f1eec108b80 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.841+0000 7f1ef0e00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec112fb0 con 0x7f1eec102b00 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1ee9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec1034c0 0x7f1eec0752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:13.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1ee9400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec1034c0 0x7f1eec0752f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55068/0 (socket says 172.21.15.16:55068) 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1ee9e00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eec102b00 0x7f1eec074db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1ee9400700 1 -- 172.21.15.16:0/3407500470 learned_addr learned my addr 172.21.15.16:0/3407500470 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1eea800700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1eec108b80 0x7f1eec075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1ee9e00700 1 -- 172.21.15.16:0/3407500470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1eec108b80 msgr2=0x7f1eec075830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1ee9e00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1eec108b80 0x7f1eec075830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1ee9e00700 1 -- 172.21.15.16:0/3407500470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec1034c0 msgr2=0x7f1eec0752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1ee9e00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec1034c0 0x7f1eec0752f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.842+0000 7f1ee9e00700 1 -- 172.21.15.16:0/3407500470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eec1a7790 con 0x7f1eec102b00 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.843+0000 7f1ee9e00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eec102b00 0x7f1eec074db0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1edc000c00 tx=0x7f1edc00fb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.843+0000 7f1ee2a00700 1 -- 172.21.15.16:0/3407500470 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc005140 con 0x7f1eec102b00 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.843+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eec1a7a20 con 0x7f1eec102b00 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.843+0000 7f1ee2a00700 1 -- 172.21.15.16:0/3407500470 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1edc00b560 con 0x7f1eec102b00 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.843+0000 7f1ee2a00700 1 -- 172.21.15.16:0/3407500470 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc003b40 con 0x7f1eec102b00 2024-04-13T03:33:13.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.843+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eec1a7f50 con 0x7f1eec102b00 2024-04-13T03:33:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.844+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eec0697a0 con 0x7f1eec102b00 2024-04-13T03:33:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.846+0000 7f1ee2a00700 1 -- 172.21.15.16:0/3407500470 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1edc00baa0 con 0x7f1eec102b00 2024-04-13T03:33:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.848+0000 7f1ee2a00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ed00a3cc0 0x7f1ed00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.848+0000 7f1ee2a00700 1 -- 172.21.15.16:0/3407500470 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f1edc0d15f0 con 0x7f1eec102b00 2024-04-13T03:33:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.848+0000 7f1ee9400700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ed00a3cc0 0x7f1ed00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:13.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.849+0000 7f1ee9400700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ed00a3cc0 0x7f1ed00a6180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1eec07a1f0 tx=0x7f1ed8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:13.808 INFO:teuthology.orchestra.run.smithi016.stdout:107374182478 2024-04-13T03:33:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:12.852+0000 7f1ee2a00700 1 -- 172.21.15.16:0/3407500470 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1edc013030 con 0x7f1eec102b00 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.133+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1eec02dfa0 con 0x7f1eec102b00 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.134+0000 7f1ee2a00700 1 -- 172.21.15.16:0/3407500470 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1edc09dca0 con 0x7f1eec102b00 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ed00a3cc0 msgr2=0x7f1ed00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ed00a3cc0 0x7f1ed00a6180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1eec07a1f0 tx=0x7f1ed8009310 comp rx=0 tx=0).stop 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eec102b00 msgr2=0x7f1eec074db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eec102b00 0x7f1eec074db0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1edc000c00 tx=0x7f1edc00fb40 comp rx=0 tx=0).stop 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 shutdown_connections 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1ed00a3cc0 0x7f1ed00a6180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1eec108b80 0x7f1eec075830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1eec102b00 0x7f1eec074db0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 --2- 172.21.15.16:0/3407500470 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec1034c0 0x7f1eec0752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.136+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 >> 172.21.15.16:0/3407500470 conn(0x7f1eec079c60 msgr2=0x7f1eec1016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.137+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 shutdown_connections 2024-04-13T03:33:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:13.137+0000 7f1ef0e00700 1 -- 172.21.15.16:0/3407500470 wait complete. 2024-04-13T03:33:14.063 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:14.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:14 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3407500470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-13T03:33:14.287 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:14.288 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:14 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3407500470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-13T03:33:14.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:14 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3407500470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-13T03:33:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:15 smithi119 ceph-mon[35835]: pgmap v369: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-13T03:33:15.339 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:15 smithi016 ceph-mon[31772]: pgmap v369: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-13T03:33:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:15 smithi188 ceph-mon[36398]: pgmap v369: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-13T03:33:15.948 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332147 got 219043332152 for osd.7 2024-04-13T03:33:15.949 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:16.033 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351003 got 347892351009 for osd.11 2024-04-13T03:33:16.034 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:17.270 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:17 smithi016 ceph-mon[31772]: pgmap v370: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-13T03:33:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:17 smithi119 ceph-mon[35835]: pgmap v370: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-13T03:33:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:17 smithi188 ceph-mon[36398]: pgmap v370: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-13T03:33:17.748 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.075+0000 7fa7db600700 1 -- 172.21.15.16:0/2211173652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 msgr2=0x7fa7d4102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.075+0000 7fa7db600700 1 --2- 172.21.15.16:0/2211173652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 0x7fa7d4102fc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009990 tx=0x7fa7c0009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.075+0000 7fa7db600700 1 -- 172.21.15.16:0/2211173652 shutdown_connections 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.075+0000 7fa7db600700 1 --2- 172.21.15.16:0/2211173652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7d4103500 0x7fa7d410f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.075+0000 7fa7db600700 1 --2- 172.21.15.16:0/2211173652 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d4108be0 0x7fa7d4108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.075+0000 7fa7db600700 1 --2- 172.21.15.16:0/2211173652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 0x7fa7d4102fc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.075+0000 7fa7db600700 1 -- 172.21.15.16:0/2211173652 >> 172.21.15.16:0/2211173652 conn(0x7fa7d4079ca0 msgr2=0x7fa7d407a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.075+0000 7fa7db600700 1 -- 172.21.15.16:0/2211173652 shutdown_connections 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.075+0000 7fa7db600700 1 -- 172.21.15.16:0/2211173652 wait complete. 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.076+0000 7fa7db600700 1 Processor -- start 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7db600700 1 -- start start 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7db600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 0x7fa7d419b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7db600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d4103500 0x7fa7d419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7db600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7d4108be0 0x7fa7d419fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7db600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d4112eb0 con 0x7fa7d4102b40 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7db600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d4112d30 con 0x7fa7d4103500 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7db600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d4113030 con 0x7fa7d4108be0 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7d8c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 0x7fa7d419b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7d8c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 0x7fa7d419b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55088/0 (socket says 172.21.15.16:55088) 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7d8c00700 1 -- 172.21.15.16:0/3134018360 learned_addr learned my addr 172.21.15.16:0/3134018360 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7d9600700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7d4108be0 0x7fa7d419fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.077+0000 7fa7d3e00700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d4103500 0x7fa7d419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7d8c00700 1 -- 172.21.15.16:0/3134018360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7d4108be0 msgr2=0x7fa7d419fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7d8c00700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7d4108be0 0x7fa7d419fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7d8c00700 1 -- 172.21.15.16:0/3134018360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d4103500 msgr2=0x7fa7d419bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7d8c00700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d4103500 0x7fa7d419bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7d8c00700 1 -- 172.21.15.16:0/3134018360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d41a05d0 con 0x7fa7d4102b40 2024-04-13T03:33:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7d8c00700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 0x7fa7d419b5c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800e9a0 tx=0x7fa7c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7d1600700 1 -- 172.21.15.16:0/3134018360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c800ca10 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7d1600700 1 -- 172.21.15.16:0/3134018360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7c800cbb0 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7d1600700 1 -- 172.21.15.16:0/3134018360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8018680 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.078+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d41a08c0 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.079+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d41a8140 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.081+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d40697a0 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.083+0000 7fa7d1600700 1 -- 172.21.15.16:0/3134018360 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa7c8010430 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.084+0000 7fa7d1600700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa7bc0a3cc0 0x7fa7bc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.084+0000 7fa7d1600700 1 -- 172.21.15.16:0/3134018360 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fa7c80d08f0 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.084+0000 7fa7d3e00700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa7bc0a3cc0 0x7fa7bc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.084+0000 7fa7d3e00700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa7bc0a3cc0 0x7fa7bc0a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa7d407a680 tx=0x7fa7c0005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.089+0000 7fa7d1600700 1 -- 172.21.15.16:0/3134018360 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7c809cf10 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.329+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fa7d41a8310 con 0x7fa7d4102b40 2024-04-13T03:33:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.330+0000 7fa7d1600700 1 -- 172.21.15.16:0/3134018360 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa7c809c810 con 0x7fa7d4102b40 2024-04-13T03:33:18.652 INFO:teuthology.orchestra.run.smithi016.stdout:313532612647 2024-04-13T03:33:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa7bc0a3cc0 msgr2=0x7fa7bc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa7bc0a3cc0 0x7fa7bc0a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa7d407a680 tx=0x7fa7c0005a90 comp rx=0 tx=0).stop 2024-04-13T03:33:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 msgr2=0x7fa7d419b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 0x7fa7d419b5c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800e9a0 tx=0x7fa7c800ecb0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 shutdown_connections 2024-04-13T03:33:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa7bc0a3cc0 0x7fa7bc0a6180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa7d4108be0 0x7fa7d419fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7d4102b40 0x7fa7d419b5c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 --2- 172.21.15.16:0/3134018360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d4103500 0x7fa7d419bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 >> 172.21.15.16:0/3134018360 conn(0x7fa7d4079ca0 msgr2=0x7fa7d410dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 shutdown_connections 2024-04-13T03:33:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:18.333+0000 7fa7db600700 1 -- 172.21.15.16:0/3134018360 wait complete. 2024-04-13T03:33:19.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:19 smithi016 ceph-mon[31772]: pgmap v371: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-13T03:33:19.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:19 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3134018360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-13T03:33:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:19 smithi119 ceph-mon[35835]: pgmap v371: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-13T03:33:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:19 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3134018360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-13T03:33:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:19 smithi188 ceph-mon[36398]: pgmap v371: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-13T03:33:19.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:19 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3134018360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-13T03:33:20.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.597+0000 7f393aa00700 1 -- 172.21.15.16:0/3664452104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934108ac0 msgr2=0x7f3934108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.597+0000 7f393aa00700 1 --2- 172.21.15.16:0/3664452104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934108ac0 0x7f3934108ea0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f39240098f0 tx=0x7f3924009c00 comp rx=0 tx=0).stop 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.598+0000 7f393aa00700 1 -- 172.21.15.16:0/3664452104 shutdown_connections 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.598+0000 7f393aa00700 1 --2- 172.21.15.16:0/3664452104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39341034c0 0x7f393410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.598+0000 7f393aa00700 1 --2- 172.21.15.16:0/3664452104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934108ac0 0x7f3934108ea0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.598+0000 7f393aa00700 1 --2- 172.21.15.16:0/3664452104 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934102b00 0x7f3934102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.598+0000 7f393aa00700 1 -- 172.21.15.16:0/3664452104 >> 172.21.15.16:0/3664452104 conn(0x7f3934079c10 msgr2=0x7f393407a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.598+0000 7f393aa00700 1 -- 172.21.15.16:0/3664452104 shutdown_connections 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.598+0000 7f393aa00700 1 -- 172.21.15.16:0/3664452104 wait complete. 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.599+0000 7f393aa00700 1 Processor -- start 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.599+0000 7f393aa00700 1 -- start start 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.599+0000 7f393aa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934102b00 0x7f393419b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f393aa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39341034c0 0x7f393419ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f393aa00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934108ac0 0x7f393419fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f393aa00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3934112e10 con 0x7f3934102b00 2024-04-13T03:33:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f393aa00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3934112c90 con 0x7f39341034c0 2024-04-13T03:33:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f393aa00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3934112f90 con 0x7f3934108ac0 2024-04-13T03:33:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f3933e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934102b00 0x7f393419b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f3933e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934102b00 0x7f393419b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54874/0 (socket says 172.21.15.16:54874) 2024-04-13T03:33:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f3933e00700 1 -- 172.21.15.16:0/184216424 learned_addr learned my addr 172.21.15.16:0/184216424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f3933400700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39341034c0 0x7f393419ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.600+0000 7f3938a00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934108ac0 0x7f393419fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:20.055 INFO:teuthology.orchestra.run.smithi016.stdout:163208757316 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3933400700 1 -- 172.21.15.16:0/184216424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934108ac0 msgr2=0x7f393419fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3933400700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934108ac0 0x7f393419fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3933400700 1 -- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934102b00 msgr2=0x7f393419b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3933400700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934102b00 0x7f393419b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3933400700 1 -- 172.21.15.16:0/184216424 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39341a04f0 con 0x7f39341034c0 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3933e00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934102b00 0x7f393419b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3938a00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934108ac0 0x7f393419fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3933400700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39341034c0 0x7f393419ba20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f392000e9f0 tx=0x7f392000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3930c00700 1 -- 172.21.15.16:0/184216424 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392000cb70 con 0x7f39341034c0 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39341a07e0 con 0x7f39341034c0 2024-04-13T03:33:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3930c00700 1 -- 172.21.15.16:0/184216424 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f392000ee90 con 0x7f39341034c0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f3930c00700 1 -- 172.21.15.16:0/184216424 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920003dc0 con 0x7f39341034c0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.601+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39341a8060 con 0x7f39341034c0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.603+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39340697a0 con 0x7f39341034c0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.604+0000 7f3930c00700 1 -- 172.21.15.16:0/184216424 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3920004000 con 0x7f39341034c0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.606+0000 7f3930c00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39180a3c40 0x7f39180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.606+0000 7f3930c00700 1 -- 172.21.15.16:0/184216424 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f39200d08e0 con 0x7f39341034c0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.606+0000 7f3933e00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39180a3c40 0x7f39180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.607+0000 7f3933e00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39180a3c40 0x7f39180a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f392400c010 tx=0x7f3924019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.610+0000 7f3930c00700 1 -- 172.21.15.16:0/184216424 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f392009cf00 con 0x7f39341034c0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.846+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f39341a8290 con 0x7f39341034c0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.846+0000 7f3930c00700 1 -- 172.21.15.16:0/184216424 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f392009c800 con 0x7f39341034c0 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39180a3c40 msgr2=0x7f39180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39180a3c40 0x7f39180a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f392400c010 tx=0x7f3924019040 comp rx=0 tx=0).stop 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39341034c0 msgr2=0x7f393419ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:20.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39341034c0 0x7f393419ba20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f392000e9f0 tx=0x7f392000ed00 comp rx=0 tx=0).stop 2024-04-13T03:33:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 shutdown_connections 2024-04-13T03:33:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f39180a3c40 0x7f39180a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3934108ac0 0x7f393419fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3934102b00 0x7f393419b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 --2- 172.21.15.16:0/184216424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39341034c0 0x7f393419ba20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 >> 172.21.15.16:0/184216424 conn(0x7f3934079c10 msgr2=0x7f393410df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 shutdown_connections 2024-04-13T03:33:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:19.850+0000 7f393aa00700 1 -- 172.21.15.16:0/184216424 wait complete. 2024-04-13T03:33:20.341 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:20 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/184216424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-13T03:33:20.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:20 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/184216424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-13T03:33:20.400 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182473 got 107374182478 for osd.3 2024-04-13T03:33:20.400 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:20.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:20 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/184216424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-13T03:33:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.227+0000 7f2f08a00700 1 -- 172.21.15.16:0/903943710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f0406b1e0 msgr2=0x7f2f0406b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.227+0000 7f2f08a00700 1 --2- 172.21.15.16:0/903943710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f0406b1e0 0x7f2f0406b640 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0009990 tx=0x7f2ef0009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.227+0000 7f2f08a00700 1 -- 172.21.15.16:0/903943710 shutdown_connections 2024-04-13T03:33:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.227+0000 7f2f08a00700 1 --2- 172.21.15.16:0/903943710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0406bb80 0x7f2f0410f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.227+0000 7f2f08a00700 1 --2- 172.21.15.16:0/903943710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f04108aa0 0x7f2f04108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.227+0000 7f2f08a00700 1 --2- 172.21.15.16:0/903943710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f0406b1e0 0x7f2f0406b640 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.227+0000 7f2f08a00700 1 -- 172.21.15.16:0/903943710 >> 172.21.15.16:0/903943710 conn(0x7f2f04078960 msgr2=0x7f2f04078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.227+0000 7f2f08a00700 1 -- 172.21.15.16:0/903943710 shutdown_connections 2024-04-13T03:33:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.227+0000 7f2f08a00700 1 -- 172.21.15.16:0/903943710 wait complete. 2024-04-13T03:33:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.228+0000 7f2f08a00700 1 Processor -- start 2024-04-13T03:33:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f08a00700 1 -- start start 2024-04-13T03:33:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f08a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0406b1e0 0x7f2f0419b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f08a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f0406bb80 0x7f2f0419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f08a00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f04108aa0 0x7f2f0419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f08a00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2f04112db0 con 0x7f2f0406b1e0 2024-04-13T03:33:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f08a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2f04112c30 con 0x7f2f0406bb80 2024-04-13T03:33:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f08a00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f2f04112f30 con 0x7f2f04108aa0 2024-04-13T03:33:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f01e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0406b1e0 0x7f2f0419b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f01400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f0406bb80 0x7f2f0419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f01e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0406b1e0 0x7f2f0419b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54900/0 (socket says 172.21.15.16:54900) 2024-04-13T03:33:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.229+0000 7f2f01e00700 1 -- 172.21.15.16:0/2493869516 learned_addr learned my addr 172.21.15.16:0/2493869516 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.230+0000 7f2f02800700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f04108aa0 0x7f2f0419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.230+0000 7f2f01400700 1 -- 172.21.15.16:0/2493869516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f04108aa0 msgr2=0x7f2f0419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.230+0000 7f2f01400700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f04108aa0 0x7f2f0419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.230+0000 7f2f01400700 1 -- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0406b1e0 msgr2=0x7f2f0419b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.230+0000 7f2f01400700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0406b1e0 0x7f2f0419b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.230+0000 7f2f01400700 1 -- 172.21.15.16:0/2493869516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f041a0420 con 0x7f2f0406bb80 2024-04-13T03:33:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.230+0000 7f2f01e00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0406b1e0 0x7f2f0419b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.230+0000 7f2f01400700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f0406bb80 0x7f2f0419b9a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0006010 tx=0x7f2ef000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.231+0000 7f2efaa00700 1 -- 172.21.15.16:0/2493869516 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0027070 con 0x7f2f0406bb80 2024-04-13T03:33:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.231+0000 7f2efaa00700 1 -- 172.21.15.16:0/2493869516 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ef00056d0 con 0x7f2f0406bb80 2024-04-13T03:33:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.231+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f041a06b0 con 0x7f2f0406bb80 2024-04-13T03:33:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.231+0000 7f2efaa00700 1 -- 172.21.15.16:0/2493869516 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0022d90 con 0x7f2f0406bb80 2024-04-13T03:33:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.231+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f041a7f90 con 0x7f2f0406bb80 2024-04-13T03:33:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.232+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f040697a0 con 0x7f2f0406bb80 2024-04-13T03:33:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.234+0000 7f2efaa00700 1 -- 172.21.15.16:0/2493869516 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2ef002a070 con 0x7f2f0406bb80 2024-04-13T03:33:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.236+0000 7f2efaa00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2ee80a3cc0 0x7f2ee80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.236+0000 7f2efaa00700 1 -- 172.21.15.16:0/2493869516 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f2ef00d1a90 con 0x7f2f0406bb80 2024-04-13T03:33:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.236+0000 7f2f01e00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2ee80a3cc0 0x7f2ee80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.237+0000 7f2f01e00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2ee80a3cc0 0x7f2ee80a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009680 tx=0x7f2ef4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:21.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.238+0000 7f2efaa00700 1 -- 172.21.15.16:0/2493869516 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ef0013070 con 0x7f2f0406bb80 2024-04-13T03:33:21.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:21 smithi016 ceph-mon[31772]: pgmap v372: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-13T03:33:21.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:21 smithi188 ceph-mon[36398]: pgmap v372: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-13T03:33:21.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.477+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f2f041a8160 con 0x7f2f0406bb80 2024-04-13T03:33:21.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.477+0000 7f2efaa00700 1 -- 172.21.15.16:0/2493869516 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2ef009e140 con 0x7f2f0406bb80 2024-04-13T03:33:21.478 INFO:teuthology.orchestra.run.smithi016.stdout:77309411415 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.481+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2ee80a3cc0 msgr2=0x7f2ee80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.481+0000 7f2f08a00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2ee80a3cc0 0x7f2ee80a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009680 tx=0x7f2ef4009260 comp rx=0 tx=0).stop 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.481+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f0406bb80 msgr2=0x7f2f0419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.481+0000 7f2f08a00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f0406bb80 0x7f2f0419b9a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0006010 tx=0x7f2ef000bab0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.481+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 shutdown_connections 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.482+0000 7f2f08a00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f2ee80a3cc0 0x7f2ee80a6180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.482+0000 7f2f08a00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f04108aa0 0x7f2f0419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.482+0000 7f2f08a00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f0406b1e0 0x7f2f0419b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.482+0000 7f2f08a00700 1 --2- 172.21.15.16:0/2493869516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f0406bb80 0x7f2f0419b9a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.482+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 >> 172.21.15.16:0/2493869516 conn(0x7f2f04078960 msgr2=0x7f2f040ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:21.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.482+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 shutdown_connections 2024-04-13T03:33:21.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.482+0000 7f2f08a00700 1 -- 172.21.15.16:0/2493869516 wait complete. 2024-04-13T03:33:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.543+0000 7fdadec00700 1 -- 172.21.15.16:0/3206151207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdad8103c00 msgr2=0x7fdad8103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.543+0000 7fdadec00700 1 --2- 172.21.15.16:0/3206151207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdad8103c00 0x7fdad8103fe0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009990 tx=0x7fdacc009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.544+0000 7fdadec00700 1 -- 172.21.15.16:0/3206151207 shutdown_connections 2024-04-13T03:33:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.544+0000 7fdadec00700 1 --2- 172.21.15.16:0/3206151207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdad810cfe0 0x7fdad810f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.544+0000 7fdadec00700 1 --2- 172.21.15.16:0/3206151207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdad8103c00 0x7fdad8103fe0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.544+0000 7fdadec00700 1 --2- 172.21.15.16:0/3206151207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdad8104520 0x7fdad810ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.544+0000 7fdadec00700 1 -- 172.21.15.16:0/3206151207 >> 172.21.15.16:0/3206151207 conn(0x7fdad80fd3d0 msgr2=0x7fdad80ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.544+0000 7fdadec00700 1 -- 172.21.15.16:0/3206151207 shutdown_connections 2024-04-13T03:33:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.544+0000 7fdadec00700 1 -- 172.21.15.16:0/3206151207 wait complete. 2024-04-13T03:33:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.545+0000 7fdadec00700 1 Processor -- start 2024-04-13T03:33:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.545+0000 7fdadec00700 1 -- start start 2024-04-13T03:33:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.545+0000 7fdadec00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdad8103c00 0x7fdad8101860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.545+0000 7fdadec00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdad8104520 0x7fdad8101da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.545+0000 7fdadec00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdad810cfe0 0x7fdad81022e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.545+0000 7fdadec00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdad8112d70 con 0x7fdad8104520 2024-04-13T03:33:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.545+0000 7fdadec00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdad8112bf0 con 0x7fdad810cfe0 2024-04-13T03:33:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.545+0000 7fdadec00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdad8112ef0 con 0x7fdad8103c00 2024-04-13T03:33:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdad8104520 0x7fdad8101da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadd800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdad8103c00 0x7fdad8101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadce00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdad8104520 0x7fdad8101da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54914/0 (socket says 172.21.15.16:54914) 2024-04-13T03:33:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadce00700 1 -- 172.21.15.16:0/3134405133 learned_addr learned my addr 172.21.15.16:0/3134405133 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdade200700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdad810cfe0 0x7fdad81022e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadce00700 1 -- 172.21.15.16:0/3134405133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdad8103c00 msgr2=0x7fdad8101860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadce00700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdad8103c00 0x7fdad8101860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadce00700 1 -- 172.21.15.16:0/3134405133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdad810cfe0 msgr2=0x7fdad81022e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadce00700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdad810cfe0 0x7fdad81022e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadce00700 1 -- 172.21.15.16:0/3134405133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad81a7730 con 0x7fdad8104520 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.546+0000 7fdadd800700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdad8103c00 0x7fdad8101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.547+0000 7fdadce00700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdad8104520 0x7fdad8101da0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdac400e9f0 tx=0x7fdac400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.547+0000 7fdad6000700 1 -- 172.21.15.16:0/3134405133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4004a30 con 0x7fdad8104520 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.547+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad81a7a20 con 0x7fdad8104520 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.547+0000 7fdad6000700 1 -- 172.21.15.16:0/3134405133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdac400cca0 con 0x7fdad8104520 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.547+0000 7fdad6000700 1 -- 172.21.15.16:0/3134405133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac40056c0 con 0x7fdad8104520 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.547+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad81a7f60 con 0x7fdad8104520 2024-04-13T03:33:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.548+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdad80697a0 con 0x7fdad8104520 2024-04-13T03:33:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.551+0000 7fdad6000700 1 -- 172.21.15.16:0/3134405133 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdac400c7c0 con 0x7fdad8104520 2024-04-13T03:33:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.552+0000 7fdad6000700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdac00a3cc0 0x7fdac00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.552+0000 7fdad6000700 1 -- 172.21.15.16:0/3134405133 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fdac40d0ac0 con 0x7fdad8104520 2024-04-13T03:33:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.552+0000 7fdadd800700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdac00a3cc0 0x7fdac00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.553+0000 7fdadd800700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdac00a3cc0 0x7fdac00a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00b2e0 tx=0x7fdacc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.555+0000 7fdad6000700 1 -- 172.21.15.16:0/3134405133 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdac4053ff0 con 0x7fdad8104520 2024-04-13T03:33:21.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:21 smithi119 ceph-mon[35835]: pgmap v372: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-13T03:33:21.746 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612641 got 313532612647 for osd.10 2024-04-13T03:33:21.746 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:21.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.800+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fdad8102d90 con 0x7fdad8104520 2024-04-13T03:33:21.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.800+0000 7fdad6000700 1 -- 172.21.15.16:0/3134405133 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdac409d0e0 con 0x7fdad8104520 2024-04-13T03:33:21.801 INFO:teuthology.orchestra.run.smithi016.stdout:141733920841 2024-04-13T03:33:21.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdac00a3cc0 msgr2=0x7fdac00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdac00a3cc0 0x7fdac00a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00b2e0 tx=0x7fdacc005fd0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdad8104520 msgr2=0x7fdad8101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdad8104520 0x7fdad8101da0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdac400e9f0 tx=0x7fdac400ed00 comp rx=0 tx=0).stop 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 shutdown_connections 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fdac00a3cc0 0x7fdac00a6180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdad810cfe0 0x7fdad81022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdad8103c00 0x7fdad8101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 --2- 172.21.15.16:0/3134405133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdad8104520 0x7fdad8101da0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.803+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 >> 172.21.15.16:0/3134405133 conn(0x7fdad80fd3d0 msgr2=0x7fdad80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.804+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 shutdown_connections 2024-04-13T03:33:21.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:21.804+0000 7fdadec00700 1 -- 172.21.15.16:0/3134405133 wait complete. 2024-04-13T03:33:22.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:22 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2493869516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-13T03:33:22.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:22 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3134405133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-13T03:33:22.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:22 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2493869516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-13T03:33:22.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:22 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3134405133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-13T03:33:22.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:22 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2493869516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-13T03:33:22.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:22 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3134405133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-13T03:33:22.610 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495671 got 197568495677 for osd.6 2024-04-13T03:33:22.610 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:23.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:23 smithi016 ceph-mon[31772]: pgmap v373: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-13T03:33:23.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:23 smithi188 ceph-mon[36398]: pgmap v373: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-13T03:33:23.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:23 smithi119 ceph-mon[35835]: pgmap v373: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-13T03:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.880+0000 7fb4b7c00700 1 -- 172.21.15.16:0/514201611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4b0108aa0 msgr2=0x7fb4b0108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.880+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/514201611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4b0108aa0 0x7fb4b0108e80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009990 tx=0x7fb4a4009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.880+0000 7fb4b7c00700 1 -- 172.21.15.16:0/514201611 shutdown_connections 2024-04-13T03:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.880+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/514201611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4b006bb80 0x7fb4b010f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:23.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.880+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/514201611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4b0108aa0 0x7fb4b0108e80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.880+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/514201611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b006b1e0 0x7fb4b006b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.880+0000 7fb4b7c00700 1 -- 172.21.15.16:0/514201611 >> 172.21.15.16:0/514201611 conn(0x7fb4b0078960 msgr2=0x7fb4b0078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.880+0000 7fb4b7c00700 1 -- 172.21.15.16:0/514201611 shutdown_connections 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.880+0000 7fb4b7c00700 1 -- 172.21.15.16:0/514201611 wait complete. 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.881+0000 7fb4b7c00700 1 Processor -- start 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.881+0000 7fb4b7c00700 1 -- start start 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4b7c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4b006b1e0 0x7fb4b0101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4b7c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b006bb80 0x7fb4b0101ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4b7c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4b0108aa0 0x7fb4b0102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4b7c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b0112dd0 con 0x7fb4b006bb80 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4b7c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b0112c50 con 0x7fb4b0108aa0 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4b7c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b0112f50 con 0x7fb4b006b1e0 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4afe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b006bb80 0x7fb4b0101ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:23.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4afe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b006bb80 0x7fb4b0101ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54936/0 (socket says 172.21.15.16:54936) 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4afe00700 1 -- 172.21.15.16:0/2522935797 learned_addr learned my addr 172.21.15.16:0/2522935797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4b5200700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4b006b1e0 0x7fb4b0101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.882+0000 7fb4b5c00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4b0108aa0 0x7fb4b0102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4afe00700 1 -- 172.21.15.16:0/2522935797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4b006b1e0 msgr2=0x7fb4b0101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4afe00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4b006b1e0 0x7fb4b0101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4afe00700 1 -- 172.21.15.16:0/2522935797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4b0108aa0 msgr2=0x7fb4b0102530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4afe00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4b0108aa0 0x7fb4b0102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4afe00700 1 -- 172.21.15.16:0/2522935797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b01a77e0 con 0x7fb4b006bb80 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4afe00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b006bb80 0x7fb4b0101ff0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00e9a0 tx=0x7fb49c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4ad600700 1 -- 172.21.15.16:0/2522935797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00ca10 con 0x7fb4b006bb80 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4ad600700 1 -- 172.21.15.16:0/2522935797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb49c00cbb0 con 0x7fb4b006bb80 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b01a7ad0 con 0x7fb4b006bb80 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.883+0000 7fb4ad600700 1 -- 172.21.15.16:0/2522935797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c018680 con 0x7fb4b006bb80 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.884+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b01a8010 con 0x7fb4b006bb80 2024-04-13T03:33:23.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.885+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4b00697a0 con 0x7fb4b006bb80 2024-04-13T03:33:23.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.887+0000 7fb4ad600700 1 -- 172.21.15.16:0/2522935797 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb49c010430 con 0x7fb4b006bb80 2024-04-13T03:33:23.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.888+0000 7fb4ad600700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4980a3c40 0x7fb4980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:23.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.888+0000 7fb4ad600700 1 -- 172.21.15.16:0/2522935797 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fb49c0d09b0 con 0x7fb4b006bb80 2024-04-13T03:33:23.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.889+0000 7fb4b5200700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4980a3c40 0x7fb4980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:23.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.889+0000 7fb4b5200700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4980a3c40 0x7fb4980a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4006010 tx=0x7fb4a400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:23.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:23.893+0000 7fb4ad600700 1 -- 172.21.15.16:0/2522935797 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb49c09cfd0 con 0x7fb4b006bb80 2024-04-13T03:33:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.132+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb4b002e160 con 0x7fb4b006bb80 2024-04-13T03:33:24.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.133+0000 7fb4ad600700 1 -- 172.21.15.16:0/2522935797 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb49c09c8d0 con 0x7fb4b006bb80 2024-04-13T03:33:24.134 INFO:teuthology.orchestra.run.smithi016.stdout:55834574940 2024-04-13T03:33:24.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4980a3c40 msgr2=0x7fb4980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4980a3c40 0x7fb4980a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4006010 tx=0x7fb4a400b260 comp rx=0 tx=0).stop 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b006bb80 msgr2=0x7fb4b0101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b006bb80 0x7fb4b0101ff0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00e9a0 tx=0x7fb49c00ecb0 comp rx=0 tx=0).stop 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 shutdown_connections 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb4980a3c40 0x7fb4980a6100 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4b0108aa0 0x7fb4b0102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4b006b1e0 0x7fb4b0101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 --2- 172.21.15.16:0/2522935797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4b006bb80 0x7fb4b0101ff0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.136+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 >> 172.21.15.16:0/2522935797 conn(0x7fb4b0078960 msgr2=0x7fb4b00ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:24.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.137+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 shutdown_connections 2024-04-13T03:33:24.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:24.137+0000 7fb4b7c00700 1 -- 172.21.15.16:0/2522935797 wait complete. 2024-04-13T03:33:25.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:25 smithi016 ceph-mon[31772]: pgmap v374: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-13T03:33:25.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:25 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2522935797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-13T03:33:25.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:33:25.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:25 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:33:25.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:25 smithi188 ceph-mon[36398]: pgmap v374: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-13T03:33:25.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:25 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2522935797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-13T03:33:25.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:33:25.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:25 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:33:25.464 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757309 got 163208757316 for osd.5 2024-04-13T03:33:25.464 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:25 smithi119 ceph-mon[35835]: pgmap v374: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-13T03:33:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:25 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2522935797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-13T03:33:25.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:33:25.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:25 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:33:26.207 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920835 got 141733920841 for osd.4 2024-04-13T03:33:26.207 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:26.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:33:26.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:33:26.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:33:26.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:33:26.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:33:26.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:33:26.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:33:26.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:33:26.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:33:26.579 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574934 got 55834574940 for osd.1 2024-04-13T03:33:26.579 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:27.183 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:27 smithi016 ceph-mon[31772]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-13T03:33:27.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:27 smithi188 ceph-mon[36398]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-13T03:33:27.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:27 smithi119 ceph-mon[35835]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-13T03:33:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.710+0000 7fac7ba00700 1 -- 172.21.15.16:0/2333238415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac741034c0 msgr2=0x7fac7410f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:27.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.710+0000 7fac7ba00700 1 --2- 172.21.15.16:0/2333238415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac741034c0 0x7fac7410f950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00b0a0 tx=0x7fac6c00b3b0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.710+0000 7fac7ba00700 1 -- 172.21.15.16:0/2333238415 shutdown_connections 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.710+0000 7fac7ba00700 1 --2- 172.21.15.16:0/2333238415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac741034c0 0x7fac7410f950 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.710+0000 7fac7ba00700 1 --2- 172.21.15.16:0/2333238415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac74108b80 0x7fac74108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.710+0000 7fac7ba00700 1 --2- 172.21.15.16:0/2333238415 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac74102b00 0x7fac74102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.710+0000 7fac7ba00700 1 -- 172.21.15.16:0/2333238415 >> 172.21.15.16:0/2333238415 conn(0x7fac74079c60 msgr2=0x7fac7407a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.710+0000 7fac7ba00700 1 -- 172.21.15.16:0/2333238415 shutdown_connections 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.710+0000 7fac7ba00700 1 -- 172.21.15.16:0/2333238415 wait complete. 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.711+0000 7fac7ba00700 1 Processor -- start 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.711+0000 7fac7ba00700 1 -- start start 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.711+0000 7fac7ba00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac74102b00 0x7fac7419b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.711+0000 7fac7ba00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac741034c0 0x7fac7419ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.711+0000 7fac7ba00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac74108b80 0x7fac7419fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac7ba00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fac74112df0 con 0x7fac74108b80 2024-04-13T03:33:27.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac7ba00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fac74112c70 con 0x7fac741034c0 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac7ba00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fac74112f70 con 0x7fac74102b00 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac74102b00 0x7fac7419b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79a00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac74108b80 0x7fac7419fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79000700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac74102b00 0x7fac7419b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:47000/0 (socket says 172.21.15.16:47000) 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79000700 1 -- 172.21.15.16:0/3844579238 learned_addr learned my addr 172.21.15.16:0/3844579238 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac73e00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac741034c0 0x7fac7419ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79000700 1 -- 172.21.15.16:0/3844579238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac741034c0 msgr2=0x7fac7419ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79000700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac741034c0 0x7fac7419ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79000700 1 -- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac74108b80 msgr2=0x7fac7419fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79000700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac74108b80 0x7fac7419fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79000700 1 -- 172.21.15.16:0/3844579238 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac741a04f0 con 0x7fac74102b00 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac79a00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac74108b80 0x7fac7419fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.712+0000 7fac73e00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac741034c0 0x7fac7419ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.713+0000 7fac79000700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac74102b00 0x7fac7419b4e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fac6800e9a0 tx=0x7fac6800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.713+0000 7fac71600700 1 -- 172.21.15.16:0/3844579238 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6800ca10 con 0x7fac74102b00 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.713+0000 7fac71600700 1 -- 172.21.15.16:0/3844579238 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac6800cbb0 con 0x7fac74102b00 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.713+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac741a07e0 con 0x7fac74102b00 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.713+0000 7fac71600700 1 -- 172.21.15.16:0/3844579238 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac68018680 con 0x7fac74102b00 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.713+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac741a8060 con 0x7fac74102b00 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.714+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac740697a0 con 0x7fac74102b00 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.717+0000 7fac71600700 1 -- 172.21.15.16:0/3844579238 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fac68010430 con 0x7fac74102b00 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.718+0000 7fac71600700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fac5c0a3cc0 0x7fac5c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.718+0000 7fac71600700 1 -- 172.21.15.16:0/3844579238 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fac680d08c0 con 0x7fac74102b00 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.719+0000 7fac73e00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fac5c0a3cc0 0x7fac5c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.719+0000 7fac73e00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fac5c0a3cc0 0x7fac5c0a6180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fac74076d60 tx=0x7fac64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:27.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.721+0000 7fac71600700 1 -- 172.21.15.16:0/3844579238 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac6809cee0 con 0x7fac74102b00 2024-04-13T03:33:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.968+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fac741a8290 con 0x7fac74102b00 2024-04-13T03:33:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.969+0000 7fac71600700 1 -- 172.21.15.16:0/3844579238 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fac6809c7e0 con 0x7fac74102b00 2024-04-13T03:33:27.970 INFO:teuthology.orchestra.run.smithi016.stdout:287762808878 2024-04-13T03:33:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fac5c0a3cc0 msgr2=0x7fac5c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fac5c0a3cc0 0x7fac5c0a6180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fac74076d60 tx=0x7fac64009260 comp rx=0 tx=0).stop 2024-04-13T03:33:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac74102b00 msgr2=0x7fac7419b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac74102b00 0x7fac7419b4e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fac6800e9a0 tx=0x7fac6800ecb0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 shutdown_connections 2024-04-13T03:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fac5c0a3cc0 0x7fac5c0a6180 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac74108b80 0x7fac7419fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac74102b00 0x7fac7419b4e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 --2- 172.21.15.16:0/3844579238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac741034c0 0x7fac7419ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 >> 172.21.15.16:0/3844579238 conn(0x7fac74079c60 msgr2=0x7fac74075d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 shutdown_connections 2024-04-13T03:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:27.972+0000 7fac7ba00700 1 -- 172.21.15.16:0/3844579238 wait complete. 2024-04-13T03:33:28.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:28 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3844579238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-13T03:33:28.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:28 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3844579238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-13T03:33:28.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:28 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3844579238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-13T03:33:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.638+0000 7f9decc00700 1 -- 172.21.15.16:0/1233069654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de81044c0 msgr2=0x7f9de810c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.638+0000 7f9decc00700 1 --2- 172.21.15.16:0/1233069654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de81044c0 0x7f9de810c9b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009990 tx=0x7f9dd0009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.638+0000 7f9decc00700 1 -- 172.21.15.16:0/1233069654 shutdown_connections 2024-04-13T03:33:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.638+0000 7f9decc00700 1 --2- 172.21.15.16:0/1233069654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9de810cfd0 0x7f9de810f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.638+0000 7f9decc00700 1 --2- 172.21.15.16:0/1233069654 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9de8103ba0 0x7f9de8103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.638+0000 7f9decc00700 1 --2- 172.21.15.16:0/1233069654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de81044c0 0x7f9de810c9b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.638+0000 7f9decc00700 1 -- 172.21.15.16:0/1233069654 >> 172.21.15.16:0/1233069654 conn(0x7f9de80fd3d0 msgr2=0x7f9de80ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.639+0000 7f9decc00700 1 -- 172.21.15.16:0/1233069654 shutdown_connections 2024-04-13T03:33:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.639+0000 7f9decc00700 1 -- 172.21.15.16:0/1233069654 wait complete. 2024-04-13T03:33:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.640+0000 7f9decc00700 1 Processor -- start 2024-04-13T03:33:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.640+0000 7f9decc00700 1 -- start start 2024-04-13T03:33:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.640+0000 7f9decc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8103ba0 0x7f9de819b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:28.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.640+0000 7f9decc00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9de81044c0 0x7f9de819b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.640+0000 7f9decc00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9de810cfd0 0x7f9de819fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9decc00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9de8112d50 con 0x7f9de8103ba0 2024-04-13T03:33:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9decc00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9de8112bd0 con 0x7f9de81044c0 2024-04-13T03:33:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9decc00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f9de8112ed0 con 0x7f9de810cfd0 2024-04-13T03:33:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9de7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8103ba0 0x7f9de819b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:28.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9de7400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8103ba0 0x7f9de819b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54984/0 (socket says 172.21.15.16:54984) 2024-04-13T03:33:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9de7400700 1 -- 172.21.15.16:0/3802521345 learned_addr learned my addr 172.21.15.16:0/3802521345 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:28.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9ddfe00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9de81044c0 0x7f9de819b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9de7e00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9de810cfd0 0x7f9de819fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9de7400700 1 -- 172.21.15.16:0/3802521345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9de810cfd0 msgr2=0x7f9de819fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9de7400700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9de810cfd0 0x7f9de819fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9de7400700 1 -- 172.21.15.16:0/3802521345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9de81044c0 msgr2=0x7f9de819b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9de7400700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9de81044c0 0x7f9de819b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.641+0000 7f9de7400700 1 -- 172.21.15.16:0/3802521345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de81a0470 con 0x7f9de8103ba0 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.642+0000 7f9de7e00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9de810cfd0 0x7f9de819fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.642+0000 7f9de7400700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8103ba0 0x7f9de819b460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800eb30 tx=0x7f9dd800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.642+0000 7f9de4c00700 1 -- 172.21.15.16:0/3802521345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd800cc90 con 0x7f9de8103ba0 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.642+0000 7f9de4c00700 1 -- 172.21.15.16:0/3802521345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dd8004500 con 0x7f9de8103ba0 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.642+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de81a0760 con 0x7f9de8103ba0 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.642+0000 7f9de4c00700 1 -- 172.21.15.16:0/3802521345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8010430 con 0x7f9de8103ba0 2024-04-13T03:33:28.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.642+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de81a7fe0 con 0x7f9de8103ba0 2024-04-13T03:33:28.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.644+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9de80697a0 con 0x7f9de8103ba0 2024-04-13T03:33:28.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.647+0000 7f9de4c00700 1 -- 172.21.15.16:0/3802521345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9dd8003f70 con 0x7f9de8103ba0 2024-04-13T03:33:28.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.649+0000 7f9de4c00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9dcc0a3c40 0x7f9dcc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:28.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.649+0000 7f9de4c00700 1 -- 172.21.15.16:0/3802521345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f9dd80d0ad0 con 0x7f9de8103ba0 2024-04-13T03:33:28.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.649+0000 7f9ddfe00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9dcc0a3c40 0x7f9dcc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:28.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.650+0000 7f9ddfe00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9dcc0a3c40 0x7f9dcc0a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9de810e830 tx=0x7f9dd000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:28.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.654+0000 7f9de4c00700 1 -- 172.21.15.16:0/3802521345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9dd809d0f0 con 0x7f9de8103ba0 2024-04-13T03:33:28.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.891+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f9de819c7c0 con 0x7f9de8103ba0 2024-04-13T03:33:28.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.891+0000 7f9de4c00700 1 -- 172.21.15.16:0/3802521345 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9dd809c9f0 con 0x7f9de8103ba0 2024-04-13T03:33:28.892 INFO:teuthology.orchestra.run.smithi016.stdout:253403070518 2024-04-13T03:33:28.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.895+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9dcc0a3c40 msgr2=0x7f9dcc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:28.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.895+0000 7f9decc00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9dcc0a3c40 0x7f9dcc0a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9de810e830 tx=0x7f9dd000b260 comp rx=0 tx=0).stop 2024-04-13T03:33:28.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.895+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8103ba0 msgr2=0x7f9de819b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:28.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.895+0000 7f9decc00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8103ba0 0x7f9de819b460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800eb30 tx=0x7f9dd800ee40 comp rx=0 tx=0).stop 2024-04-13T03:33:28.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.896+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 shutdown_connections 2024-04-13T03:33:28.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.896+0000 7f9decc00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f9dcc0a3c40 0x7f9dcc0a6100 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.896+0000 7f9decc00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9de810cfd0 0x7f9de819fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.896+0000 7f9decc00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8103ba0 0x7f9de819b460 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.896+0000 7f9decc00700 1 --2- 172.21.15.16:0/3802521345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9de81044c0 0x7f9de819b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:28.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.896+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 >> 172.21.15.16:0/3802521345 conn(0x7f9de80fd3d0 msgr2=0x7f9de80ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:28.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.896+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 shutdown_connections 2024-04-13T03:33:28.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:28.896+0000 7f9decc00700 1 -- 172.21.15.16:0/3802521345 wait complete. 2024-04-13T03:33:29.003 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411415 for osd.2 2024-04-13T03:33:29.004 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:29.306 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:29 smithi016 ceph-mon[31772]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:29.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:29 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3802521345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-13T03:33:29.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:29 smithi188 ceph-mon[36398]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:29.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:29 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3802521345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-13T03:33:29.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:29 smithi119 ceph-mon[35835]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:29.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:29 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3802521345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-13T03:33:29.829 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808873 got 287762808878 for osd.9 2024-04-13T03:33:29.830 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:30.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070513 got 253403070518 for osd.8 2024-04-13T03:33:30.306 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.643+0000 7fd8a0c00700 1 -- 172.21.15.16:0/118982168 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd89c06b160 msgr2=0x7fd89c10f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.643+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/118982168 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd89c06b160 0x7fd89c10f970 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd894009990 tx=0x7fd894009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.643+0000 7fd8a0c00700 1 -- 172.21.15.16:0/118982168 shutdown_connections 2024-04-13T03:33:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.643+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/118982168 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd89c06b160 0x7fd89c10f970 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.643+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/118982168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd89c108aa0 0x7fd89c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.643+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/118982168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89c06a7c0 0x7fd89c06ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.643+0000 7fd8a0c00700 1 -- 172.21.15.16:0/118982168 >> 172.21.15.16:0/118982168 conn(0x7fd89c078960 msgr2=0x7fd89c078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.643+0000 7fd8a0c00700 1 -- 172.21.15.16:0/118982168 shutdown_connections 2024-04-13T03:33:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.643+0000 7fd8a0c00700 1 -- 172.21.15.16:0/118982168 wait complete. 2024-04-13T03:33:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.645+0000 7fd8a0c00700 1 Processor -- start 2024-04-13T03:33:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.645+0000 7fd8a0c00700 1 -- start start 2024-04-13T03:33:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.645+0000 7fd8a0c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89c06a7c0 0x7fd89c101a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.645+0000 7fd8a0c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd89c06b160 0x7fd89c101fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.645+0000 7fd8a0c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd89c108aa0 0x7fd89c102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.645+0000 7fd8a0c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c112de0 con 0x7fd89c06a7c0 2024-04-13T03:33:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.645+0000 7fd8a0c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c112c60 con 0x7fd89c06b160 2024-04-13T03:33:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.645+0000 7fd8a0c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c112f60 con 0x7fd89c108aa0 2024-04-13T03:33:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd89c06b160 0x7fd89c101fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd89c06b160 0x7fd89c101fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:37054/0 (socket says 172.21.15.16:37054) 2024-04-13T03:33:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899400700 1 -- 172.21.15.16:0/3195303800 learned_addr learned my addr 172.21.15.16:0/3195303800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899e00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89c06a7c0 0x7fd89c101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd89a800700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd89c108aa0 0x7fd89c102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899400700 1 -- 172.21.15.16:0/3195303800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd89c108aa0 msgr2=0x7fd89c102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899400700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd89c108aa0 0x7fd89c102500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899400700 1 -- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89c06a7c0 msgr2=0x7fd89c101a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899400700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89c06a7c0 0x7fd89c101a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899400700 1 -- 172.21.15.16:0/3195303800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c1a7770 con 0x7fd89c06b160 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.646+0000 7fd899400700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd89c06b160 0x7fd89c101fc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd88800e9a0 tx=0x7fd88800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.647+0000 7fd892a00700 1 -- 172.21.15.16:0/3195303800 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88800cbd0 con 0x7fd89c06b160 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.647+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd89c1a7a60 con 0x7fd89c06b160 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.647+0000 7fd892a00700 1 -- 172.21.15.16:0/3195303800 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd88800cd70 con 0x7fd89c06b160 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.647+0000 7fd892a00700 1 -- 172.21.15.16:0/3195303800 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888018750 con 0x7fd89c06b160 2024-04-13T03:33:30.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.647+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd89c1a7ff0 con 0x7fd89c06b160 2024-04-13T03:33:30.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.648+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd89c0697a0 con 0x7fd89c06b160 2024-04-13T03:33:30.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.651+0000 7fd892a00700 1 -- 172.21.15.16:0/3195303800 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd888010430 con 0x7fd89c06b160 2024-04-13T03:33:30.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.652+0000 7fd892a00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd8800a3cc0 0x7fd8800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.652+0000 7fd892a00700 1 -- 172.21.15.16:0/3195303800 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7fd8880d09b0 con 0x7fd89c06b160 2024-04-13T03:33:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.653+0000 7fd899e00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd8800a3cc0 0x7fd8800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:30.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.653+0000 7fd899e00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd8800a3cc0 0x7fd8800a6180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00f540 tx=0x7fd88c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:30.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.654+0000 7fd892a00700 1 -- 172.21.15.16:0/3195303800 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd88809cfd0 con 0x7fd89c06b160 2024-04-13T03:33:30.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.886+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd89c02dfa0 con 0x7fd89c06b160 2024-04-13T03:33:30.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.887+0000 7fd892a00700 1 -- 172.21.15.16:0/3195303800 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd88809c8d0 con 0x7fd89c06b160 2024-04-13T03:33:30.887 INFO:teuthology.orchestra.run.smithi016.stdout:34359738466 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd8800a3cc0 msgr2=0x7fd8800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd8800a3cc0 0x7fd8800a6180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00f540 tx=0x7fd88c00b040 comp rx=0 tx=0).stop 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd89c06b160 msgr2=0x7fd89c101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd89c06b160 0x7fd89c101fc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd88800e9a0 tx=0x7fd88800ecb0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 shutdown_connections 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd8800a3cc0 0x7fd8800a6180 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd89c108aa0 0x7fd89c102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89c06a7c0 0x7fd89c101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 --2- 172.21.15.16:0/3195303800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd89c06b160 0x7fd89c101fc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.890+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 >> 172.21.15.16:0/3195303800 conn(0x7fd89c078960 msgr2=0x7fd89c0ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.891+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 shutdown_connections 2024-04-13T03:33:30.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:30.891+0000 7fd8a0c00700 1 -- 172.21.15.16:0/3195303800 wait complete. 2024-04-13T03:33:31.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:31 smithi016 ceph-mon[31772]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:33:31.266 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:31 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3195303800' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-13T03:33:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:31 smithi188 ceph-mon[36398]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:33:31.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:31 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3195303800' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-13T03:33:31.373 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:31 smithi119 ceph-mon[35835]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:33:31.373 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:31 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3195303800' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-13T03:33:31.718 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738458 got 34359738466 for osd.0 2024-04-13T03:33:31.718 DEBUG:teuthology.parallel:result is None 2024-04-13T03:33:31.718 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-13T03:33:31.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph pg dump --format=json 2024-04-13T03:33:32.192 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:33.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:33 smithi016 ceph-mon[31772]: pgmap v378: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:33.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:33 smithi119 ceph-mon[35835]: pgmap v378: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:33.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:33 smithi188 ceph-mon[36398]: pgmap v378: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.457+0000 7f88be200700 1 -- 172.21.15.16:0/13795997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b810f4b0 msgr2=0x7f88b81118a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.457+0000 7f88be200700 1 --2- 172.21.15.16:0/13795997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b810f4b0 0x7f88b81118a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f88b000b0a0 tx=0x7f88b000b3b0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.457+0000 7f88be200700 1 -- 172.21.15.16:0/13795997 shutdown_connections 2024-04-13T03:33:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.457+0000 7f88be200700 1 --2- 172.21.15.16:0/13795997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b810f4b0 0x7f88b81118a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.457+0000 7f88be200700 1 --2- 172.21.15.16:0/13795997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88b8074ce0 0x7f88b80750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.457+0000 7f88be200700 1 --2- 172.21.15.16:0/13795997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88b8075600 0x7f88b810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.457+0000 7f88be200700 1 -- 172.21.15.16:0/13795997 >> 172.21.15.16:0/13795997 conn(0x7f88b8079c10 msgr2=0x7f88b807a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.457+0000 7f88be200700 1 -- 172.21.15.16:0/13795997 shutdown_connections 2024-04-13T03:33:34.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.457+0000 7f88be200700 1 -- 172.21.15.16:0/13795997 wait complete. 2024-04-13T03:33:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.459+0000 7f88be200700 1 Processor -- start 2024-04-13T03:33:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.459+0000 7f88be200700 1 -- start start 2024-04-13T03:33:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.459+0000 7f88be200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88b8074ce0 0x7f88b819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.459+0000 7f88be200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b8075600 0x7f88b819b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.459+0000 7f88be200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88b810f4b0 0x7f88b819fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.459+0000 7f88be200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8114f90 con 0x7f88b8075600 2024-04-13T03:33:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.459+0000 7f88be200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8114e10 con 0x7f88b810f4b0 2024-04-13T03:33:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.459+0000 7f88be200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8115110 con 0x7f88b8074ce0 2024-04-13T03:33:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88b7400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88b8074ce0 0x7f88b819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88aea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b8075600 0x7f88b819b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88b7400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88b8074ce0 0x7f88b819b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:55946/0 (socket says 172.21.15.16:55946) 2024-04-13T03:33:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88b7400700 1 -- 172.21.15.16:0/5543497 learned_addr learned my addr 172.21.15.16:0/5543497 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88b7e00700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88b810f4b0 0x7f88b819fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88b7e00700 1 -- 172.21.15.16:0/5543497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88b8074ce0 msgr2=0x7f88b819b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88b7e00700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88b8074ce0 0x7f88b819b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88b7e00700 1 -- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b8075600 msgr2=0x7f88b819b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88b7e00700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b8075600 0x7f88b819b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88b7e00700 1 -- 172.21.15.16:0/5543497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b81a03b0 con 0x7f88b810f4b0 2024-04-13T03:33:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.460+0000 7f88aea00700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b8075600 0x7f88b819b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.461+0000 7f88b7e00700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88b810f4b0 0x7f88b819fc70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f88b806b340 tx=0x7f88b0004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.461+0000 7f88b4c00700 1 -- 172.21.15.16:0/5543497 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b0021070 con 0x7f88b810f4b0 2024-04-13T03:33:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.461+0000 7f88b4c00700 1 -- 172.21.15.16:0/5543497 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88b0007b80 con 0x7f88b810f4b0 2024-04-13T03:33:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.461+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b81a0640 con 0x7f88b810f4b0 2024-04-13T03:33:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.461+0000 7f88b4c00700 1 -- 172.21.15.16:0/5543497 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b00052a0 con 0x7f88b810f4b0 2024-04-13T03:33:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.461+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b81a7f20 con 0x7f88b810f4b0 2024-04-13T03:33:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.462+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b80697a0 con 0x7f88b810f4b0 2024-04-13T03:33:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.464+0000 7f88b4c00700 1 -- 172.21.15.16:0/5543497 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f88b0024070 con 0x7f88b810f4b0 2024-04-13T03:33:34.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.466+0000 7f88b4c00700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f889c0a3cc0 0x7f889c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.466+0000 7f88b4c00700 1 -- 172.21.15.16:0/5543497 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f88b00d15e0 con 0x7f88b810f4b0 2024-04-13T03:33:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.466+0000 7f88b7400700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f889c0a3cc0 0x7f889c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:34.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.467+0000 7f88b7400700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f889c0a3cc0 0x7f889c0a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f88a8009680 tx=0x7f88a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:34.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.470+0000 7f88b4c00700 1 -- 172.21.15.16:0/5543497 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88b009dc90 con 0x7f88b810f4b0 2024-04-13T03:33:34.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.698+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f88b8100350 con 0x7f889c0a3cc0 2024-04-13T03:33:34.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.719+0000 7f88b4c00700 1 -- 172.21.15.16:0/5543497 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340846 (secure 0 0 0) 0x7f88b8100350 con 0x7f889c0a3cc0 2024-04-13T03:33:34.726 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f889c0a3cc0 msgr2=0x7f889c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f889c0a3cc0 0x7f889c0a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f88a8009680 tx=0x7f88a8009260 comp rx=0 tx=0).stop 2024-04-13T03:33:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88b810f4b0 msgr2=0x7f88b819fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88b810f4b0 0x7f88b819fc70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f88b806b340 tx=0x7f88b0004d00 comp rx=0 tx=0).stop 2024-04-13T03:33:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 shutdown_connections 2024-04-13T03:33:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f889c0a3cc0 0x7f889c0a6180 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88b810f4b0 0x7f88b819fc70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88b8074ce0 0x7f88b819b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 --2- 172.21.15.16:0/5543497 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88b8075600 0x7f88b819b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 >> 172.21.15.16:0/5543497 conn(0x7f88b8079c10 msgr2=0x7f88b80ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 shutdown_connections 2024-04-13T03:33:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:34.726+0000 7f88be200700 1 -- 172.21.15.16:0/5543497 wait complete. 2024-04-13T03:33:34.730 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-13T03:33:35.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:35 smithi016 ceph-mon[31772]: pgmap v379: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:35.535 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-04-13T03:33:33.920838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662543,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1005,"num_read_kb":632,"num_write":436,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1245054,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":859,"ondisk_log_size":859,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369868,"kb_used_data":12396,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571956,"statfs":{"total":1151940427776,"available":1151561682944,"internally_reserved":0,"allocated":12693504,"data_stored":8053391,"data_compressed":370453,"data_compressed_allocated":499712,"data_compressed_original":2535424,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006004"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.890185+0000","last_change":"2024-04-13T03:31:37.597340+0000","last_active":"2024-04-13T03:33:09.890185+0000","last_peered":"2024-04-13T03:33:09.890185+0000","last_clean":"2024-04-13T03:33:09.890185+0000","last_became_active":"2024-04-13T03:31:37.597081+0000","last_became_peered":"2024-04-13T03:31:37.597081+0000","last_unstale":"2024-04-13T03:33:09.890185+0000","last_undegraded":"2024-04-13T03:33:09.890185+0000","last_fullsized":"2024-04-13T03:33:09.890185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:10:36.679921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.875017+0000","last_change":"2024-04-13T03:31:37.578150+0000","last_active":"2024-04-13T03:33:09.875017+0000","last_peered":"2024-04-13T03:33:09.875017+0000","last_clean":"2024-04-13T03:33:09.875017+0000","last_became_active":"2024-04-13T03:31:37.574207+0000","last_became_peered":"2024-04-13T03:31:37.574207+0000","last_unstale":"2024-04-13T03:33:09.875017+0000","last_undegraded":"2024-04-13T03:33:09.875017+0000","last_fullsized":"2024-04-13T03:33:09.875017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:38:41.999046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.892570+0000","last_change":"2024-04-13T03:31:37.581782+0000","last_active":"2024-04-13T03:33:09.892570+0000","last_peered":"2024-04-13T03:33:09.892570+0000","last_clean":"2024-04-13T03:33:09.892570+0000","last_became_active":"2024-04-13T03:31:37.581448+0000","last_became_peered":"2024-04-13T03:31:37.581448+0000","last_unstale":"2024-04-13T03:33:09.892570+0000","last_undegraded":"2024-04-13T03:33:09.892570+0000","last_fullsized":"2024-04-13T03:33:09.892570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:56:44.538333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.864571+0000","last_change":"2024-04-13T03:31:37.578982+0000","last_active":"2024-04-13T03:33:09.864571+0000","last_peered":"2024-04-13T03:33:09.864571+0000","last_clean":"2024-04-13T03:33:09.864571+0000","last_became_active":"2024-04-13T03:31:37.578375+0000","last_became_peered":"2024-04-13T03:31:37.578375+0000","last_unstale":"2024-04-13T03:33:09.864571+0000","last_undegraded":"2024-04-13T03:33:09.864571+0000","last_fullsized":"2024-04-13T03:33:09.864571+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:13:36.312872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.884519+0000","last_change":"2024-04-13T03:31:37.577546+0000","last_active":"2024-04-13T03:33:09.884519+0000","last_peered":"2024-04-13T03:33:09.884519+0000","last_clean":"2024-04-13T03:33:09.884519+0000","last_became_active":"2024-04-13T03:31:37.577238+0000","last_became_peered":"2024-04-13T03:31:37.577238+0000","last_unstale":"2024-04-13T03:33:09.884519+0000","last_undegraded":"2024-04-13T03:33:09.884519+0000","last_fullsized":"2024-04-13T03:33:09.884519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:25:17.759986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.869740+0000","last_change":"2024-04-13T03:31:37.585806+0000","last_active":"2024-04-13T03:33:09.869740+0000","last_peered":"2024-04-13T03:33:09.869740+0000","last_clean":"2024-04-13T03:33:09.869740+0000","last_became_active":"2024-04-13T03:31:37.585448+0000","last_became_peered":"2024-04-13T03:31:37.585448+0000","last_unstale":"2024-04-13T03:33:09.869740+0000","last_undegraded":"2024-04-13T03:33:09.869740+0000","last_fullsized":"2024-04-13T03:33:09.869740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:32:19.336091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.878071+0000","last_change":"2024-04-13T03:32:41.738725+0000","last_active":"2024-04-13T03:33:09.878071+0000","last_peered":"2024-04-13T03:33:09.878071+0000","last_clean":"2024-04-13T03:33:09.878071+0000","last_became_active":"2024-04-13T03:32:41.738281+0000","last_became_peered":"2024-04-13T03:32:41.738281+0000","last_unstale":"2024-04-13T03:33:09.878071+0000","last_undegraded":"2024-04-13T03:33:09.878071+0000","last_fullsized":"2024-04-13T03:33:09.878071+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:21:44.928608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.894855+0000","last_change":"2024-04-13T03:31:37.587374+0000","last_active":"2024-04-13T03:33:09.894855+0000","last_peered":"2024-04-13T03:33:09.894855+0000","last_clean":"2024-04-13T03:33:09.894855+0000","last_became_active":"2024-04-13T03:31:37.586998+0000","last_became_peered":"2024-04-13T03:31:37.586998+0000","last_unstale":"2024-04-13T03:33:09.894855+0000","last_undegraded":"2024-04-13T03:33:09.894855+0000","last_fullsized":"2024-04-13T03:33:09.894855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:16:46.441001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.886660+0000","last_change":"2024-04-13T03:31:37.580579+0000","last_active":"2024-04-13T03:33:09.886660+0000","last_peered":"2024-04-13T03:33:09.886660+0000","last_clean":"2024-04-13T03:33:09.886660+0000","last_became_active":"2024-04-13T03:31:37.580243+0000","last_became_peered":"2024-04-13T03:31:37.580243+0000","last_unstale":"2024-04-13T03:33:09.886660+0000","last_undegraded":"2024-04-13T03:33:09.886660+0000","last_fullsized":"2024-04-13T03:33:09.886660+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:42.080260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.868726+0000","last_change":"2024-04-13T03:31:37.567731+0000","last_active":"2024-04-13T03:33:09.868726+0000","last_peered":"2024-04-13T03:33:09.868726+0000","last_clean":"2024-04-13T03:33:09.868726+0000","last_became_active":"2024-04-13T03:31:37.567402+0000","last_became_peered":"2024-04-13T03:31:37.567402+0000","last_unstale":"2024-04-13T03:33:09.868726+0000","last_undegraded":"2024-04-13T03:33:09.868726+0000","last_fullsized":"2024-04-13T03:33:09.868726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:54:46.976698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887191+0000","last_change":"2024-04-13T03:31:37.576379+0000","last_active":"2024-04-13T03:33:09.887191+0000","last_peered":"2024-04-13T03:33:09.887191+0000","last_clean":"2024-04-13T03:33:09.887191+0000","last_became_active":"2024-04-13T03:31:37.576062+0000","last_became_peered":"2024-04-13T03:31:37.576062+0000","last_unstale":"2024-04-13T03:33:09.887191+0000","last_undegraded":"2024-04-13T03:33:09.887191+0000","last_fullsized":"2024-04-13T03:33:09.887191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:21:17.507656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887040+0000","last_change":"2024-04-13T03:32:41.737035+0000","last_active":"2024-04-13T03:33:09.887040+0000","last_peered":"2024-04-13T03:33:09.887040+0000","last_clean":"2024-04-13T03:33:09.887040+0000","last_became_active":"2024-04-13T03:32:41.736156+0000","last_became_peered":"2024-04-13T03:32:41.736156+0000","last_unstale":"2024-04-13T03:33:09.887040+0000","last_undegraded":"2024-04-13T03:33:09.887040+0000","last_fullsized":"2024-04-13T03:33:09.887040+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:29:37.192351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.891860+0000","last_change":"2024-04-13T03:31:37.576827+0000","last_active":"2024-04-13T03:33:09.891860+0000","last_peered":"2024-04-13T03:33:09.891860+0000","last_clean":"2024-04-13T03:33:09.891860+0000","last_became_active":"2024-04-13T03:31:37.576418+0000","last_became_peered":"2024-04-13T03:31:37.576418+0000","last_unstale":"2024-04-13T03:33:09.891860+0000","last_undegraded":"2024-04-13T03:33:09.891860+0000","last_fullsized":"2024-04-13T03:33:09.891860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:45:22.462714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887378+0000","last_change":"2024-04-13T03:31:37.577623+0000","last_active":"2024-04-13T03:33:09.887378+0000","last_peered":"2024-04-13T03:33:09.887378+0000","last_clean":"2024-04-13T03:33:09.887378+0000","last_became_active":"2024-04-13T03:31:37.577175+0000","last_became_peered":"2024-04-13T03:31:37.577175+0000","last_unstale":"2024-04-13T03:33:09.887378+0000","last_undegraded":"2024-04-13T03:33:09.887378+0000","last_fullsized":"2024-04-13T03:33:09.887378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:52:29.604492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.895425+0000","last_change":"2024-04-13T03:32:41.745239+0000","last_active":"2024-04-13T03:33:09.895425+0000","last_peered":"2024-04-13T03:33:09.895425+0000","last_clean":"2024-04-13T03:33:09.895425+0000","last_became_active":"2024-04-13T03:32:41.744076+0000","last_became_peered":"2024-04-13T03:32:41.744076+0000","last_unstale":"2024-04-13T03:33:09.895425+0000","last_undegraded":"2024-04-13T03:33:09.895425+0000","last_fullsized":"2024-04-13T03:33:09.895425+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:50.937028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.885580+0000","last_change":"2024-04-13T03:31:37.581073+0000","last_active":"2024-04-13T03:33:09.885580+0000","last_peered":"2024-04-13T03:33:09.885580+0000","last_clean":"2024-04-13T03:33:09.885580+0000","last_became_active":"2024-04-13T03:31:37.580719+0000","last_became_peered":"2024-04-13T03:31:37.580719+0000","last_unstale":"2024-04-13T03:33:09.885580+0000","last_undegraded":"2024-04-13T03:33:09.885580+0000","last_fullsized":"2024-04-13T03:33:09.885580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:22.542263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.890435+0000","last_change":"2024-04-13T03:31:37.597572+0000","last_active":"2024-04-13T03:33:09.890435+0000","last_peered":"2024-04-13T03:33:09.890435+0000","last_clean":"2024-04-13T03:33:09.890435+0000","last_became_active":"2024-04-13T03:31:37.597244+0000","last_became_peered":"2024-04-13T03:31:37.597244+0000","last_unstale":"2024-04-13T03:33:09.890435+0000","last_undegraded":"2024-04-13T03:33:09.890435+0000","last_fullsized":"2024-04-13T03:33:09.890435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:39:13.365427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.885576+0000","last_change":"2024-04-13T03:31:37.596749+0000","last_active":"2024-04-13T03:33:09.885576+0000","last_peered":"2024-04-13T03:33:09.885576+0000","last_clean":"2024-04-13T03:33:09.885576+0000","last_became_active":"2024-04-13T03:31:37.592409+0000","last_became_peered":"2024-04-13T03:31:37.592409+0000","last_unstale":"2024-04-13T03:33:09.885576+0000","last_undegraded":"2024-04-13T03:33:09.885576+0000","last_fullsized":"2024-04-13T03:33:09.885576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:35:22.360224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.880477+0000","last_change":"2024-04-13T03:31:37.571354+0000","last_active":"2024-04-13T03:33:09.880477+0000","last_peered":"2024-04-13T03:33:09.880477+0000","last_clean":"2024-04-13T03:33:09.880477+0000","last_became_active":"2024-04-13T03:31:37.571012+0000","last_became_peered":"2024-04-13T03:31:37.571012+0000","last_unstale":"2024-04-13T03:33:09.880477+0000","last_undegraded":"2024-04-13T03:33:09.880477+0000","last_fullsized":"2024-04-13T03:33:09.880477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:50:26.492090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.736666+0000","last_change":"2024-04-13T03:31:41.844448+0000","last_active":"2024-04-13T03:32:41.736666+0000","last_peered":"2024-04-13T03:32:41.736666+0000","last_clean":"2024-04-13T03:32:41.736666+0000","last_became_active":"2024-04-13T03:31:41.844143+0000","last_became_peered":"2024-04-13T03:31:41.844143+0000","last_unstale":"2024-04-13T03:32:41.736666+0000","last_undegraded":"2024-04-13T03:32:41.736666+0000","last_fullsized":"2024-04-13T03:32:41.736666+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:26:40.811950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":125,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:29.981942+0000","last_change":"2024-04-13T03:31:36.581372+0000","last_active":"2024-04-13T03:33:29.981942+0000","last_peered":"2024-04-13T03:33:29.981942+0000","last_clean":"2024-04-13T03:33:29.981942+0000","last_became_active":"2024-04-13T03:31:36.580599+0000","last_became_peered":"2024-04-13T03:31:36.580599+0000","last_unstale":"2024-04-13T03:33:29.981942+0000","last_undegraded":"2024-04-13T03:33:29.981942+0000","last_fullsized":"2024-04-13T03:33:29.981942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:10:09.839049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.893115+0000","last_change":"2024-04-13T03:31:37.584511+0000","last_active":"2024-04-13T03:33:09.893115+0000","last_peered":"2024-04-13T03:33:09.893115+0000","last_clean":"2024-04-13T03:33:09.893115+0000","last_became_active":"2024-04-13T03:31:37.584094+0000","last_became_peered":"2024-04-13T03:31:37.584094+0000","last_unstale":"2024-04-13T03:33:09.893115+0000","last_undegraded":"2024-04-13T03:33:09.893115+0000","last_fullsized":"2024-04-13T03:33:09.893115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:47:36.163763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728298+0000","last_change":"2024-04-13T03:31:41.834169+0000","last_active":"2024-04-13T03:32:41.728298+0000","last_peered":"2024-04-13T03:32:41.728298+0000","last_clean":"2024-04-13T03:32:41.728298+0000","last_became_active":"2024-04-13T03:31:41.833545+0000","last_became_peered":"2024-04-13T03:31:41.833545+0000","last_unstale":"2024-04-13T03:32:41.728298+0000","last_undegraded":"2024-04-13T03:32:41.728298+0000","last_fullsized":"2024-04-13T03:32:41.728298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:37:29.657978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.908699+0000","last_change":"2024-04-13T03:31:36.596225+0000","last_active":"2024-04-13T03:32:41.908699+0000","last_peered":"2024-04-13T03:32:41.908699+0000","last_clean":"2024-04-13T03:32:41.908699+0000","last_became_active":"2024-04-13T03:31:36.595520+0000","last_became_peered":"2024-04-13T03:31:36.595520+0000","last_unstale":"2024-04-13T03:32:41.908699+0000","last_undegraded":"2024-04-13T03:32:41.908699+0000","last_fullsized":"2024-04-13T03:32:41.908699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:15:35.172065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.896101+0000","last_change":"2024-04-13T03:31:37.587264+0000","last_active":"2024-04-13T03:33:09.896101+0000","last_peered":"2024-04-13T03:33:09.896101+0000","last_clean":"2024-04-13T03:33:09.896101+0000","last_became_active":"2024-04-13T03:31:37.585574+0000","last_became_peered":"2024-04-13T03:31:37.585574+0000","last_unstale":"2024-04-13T03:33:09.896101+0000","last_undegraded":"2024-04-13T03:33:09.896101+0000","last_fullsized":"2024-04-13T03:33:09.896101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:39.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728520+0000","last_change":"2024-04-13T03:31:41.827237+0000","last_active":"2024-04-13T03:32:41.728520+0000","last_peered":"2024-04-13T03:32:41.728520+0000","last_clean":"2024-04-13T03:32:41.728520+0000","last_became_active":"2024-04-13T03:31:41.826799+0000","last_became_peered":"2024-04-13T03:31:41.826799+0000","last_unstale":"2024-04-13T03:32:41.728520+0000","last_undegraded":"2024-04-13T03:32:41.728520+0000","last_fullsized":"2024-04-13T03:32:41.728520+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:26:14.337654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728623+0000","last_change":"2024-04-13T03:31:36.564752+0000","last_active":"2024-04-13T03:32:41.728623+0000","last_peered":"2024-04-13T03:32:41.728623+0000","last_clean":"2024-04-13T03:32:41.728623+0000","last_became_active":"2024-04-13T03:31:36.564413+0000","last_became_peered":"2024-04-13T03:31:36.564413+0000","last_unstale":"2024-04-13T03:32:41.728623+0000","last_undegraded":"2024-04-13T03:32:41.728623+0000","last_fullsized":"2024-04-13T03:32:41.728623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:34:52.369875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.890659+0000","last_change":"2024-04-13T03:31:37.579316+0000","last_active":"2024-04-13T03:33:09.890659+0000","last_peered":"2024-04-13T03:33:09.890659+0000","last_clean":"2024-04-13T03:33:09.890659+0000","last_became_active":"2024-04-13T03:31:37.578971+0000","last_became_peered":"2024-04-13T03:31:37.578971+0000","last_unstale":"2024-04-13T03:33:09.890659+0000","last_undegraded":"2024-04-13T03:33:09.890659+0000","last_fullsized":"2024-04-13T03:33:09.890659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:54:57.013722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728358+0000","last_change":"2024-04-13T03:31:41.841120+0000","last_active":"2024-04-13T03:32:41.728358+0000","last_peered":"2024-04-13T03:32:41.728358+0000","last_clean":"2024-04-13T03:32:41.728358+0000","last_became_active":"2024-04-13T03:31:41.840833+0000","last_became_peered":"2024-04-13T03:31:41.840833+0000","last_unstale":"2024-04-13T03:32:41.728358+0000","last_undegraded":"2024-04-13T03:32:41.728358+0000","last_fullsized":"2024-04-13T03:32:41.728358+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:13:09.979055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728913+0000","last_change":"2024-04-13T03:31:36.561428+0000","last_active":"2024-04-13T03:32:41.728913+0000","last_peered":"2024-04-13T03:32:41.728913+0000","last_clean":"2024-04-13T03:32:41.728913+0000","last_became_active":"2024-04-13T03:31:36.561100+0000","last_became_peered":"2024-04-13T03:31:36.561100+0000","last_unstale":"2024-04-13T03:32:41.728913+0000","last_undegraded":"2024-04-13T03:32:41.728913+0000","last_fullsized":"2024-04-13T03:32:41.728913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:04:32.125292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.885918+0000","last_change":"2024-04-13T03:32:41.736642+0000","last_active":"2024-04-13T03:33:09.885918+0000","last_peered":"2024-04-13T03:33:09.885918+0000","last_clean":"2024-04-13T03:33:09.885918+0000","last_became_active":"2024-04-13T03:32:41.736237+0000","last_became_peered":"2024-04-13T03:32:41.736237+0000","last_unstale":"2024-04-13T03:33:09.885918+0000","last_undegraded":"2024-04-13T03:33:09.885918+0000","last_fullsized":"2024-04-13T03:33:09.885918+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:10:14.027179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.737603+0000","last_change":"2024-04-13T03:31:41.824295+0000","last_active":"2024-04-13T03:32:41.737603+0000","last_peered":"2024-04-13T03:32:41.737603+0000","last_clean":"2024-04-13T03:32:41.737603+0000","last_became_active":"2024-04-13T03:31:41.823810+0000","last_became_peered":"2024-04-13T03:31:41.823810+0000","last_unstale":"2024-04-13T03:32:41.737603+0000","last_undegraded":"2024-04-13T03:32:41.737603+0000","last_fullsized":"2024-04-13T03:32:41.737603+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:40:08.888519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":119,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:27.979971+0000","last_change":"2024-04-13T03:31:36.558866+0000","last_active":"2024-04-13T03:33:27.979971+0000","last_peered":"2024-04-13T03:33:27.979971+0000","last_clean":"2024-04-13T03:33:27.979971+0000","last_became_active":"2024-04-13T03:31:36.558541+0000","last_became_peered":"2024-04-13T03:31:36.558541+0000","last_unstale":"2024-04-13T03:33:27.979971+0000","last_undegraded":"2024-04-13T03:33:27.979971+0000","last_fullsized":"2024-04-13T03:33:27.979971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:59:29.431084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.882838+0000","last_change":"2024-04-13T03:31:37.581298+0000","last_active":"2024-04-13T03:33:09.882838+0000","last_peered":"2024-04-13T03:33:09.882838+0000","last_clean":"2024-04-13T03:33:09.882838+0000","last_became_active":"2024-04-13T03:31:37.580963+0000","last_became_peered":"2024-04-13T03:31:37.580963+0000","last_unstale":"2024-04-13T03:33:09.882838+0000","last_undegraded":"2024-04-13T03:33:09.882838+0000","last_fullsized":"2024-04-13T03:33:09.882838+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:47:27.921928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.737546+0000","last_change":"2024-04-13T03:31:41.831221+0000","last_active":"2024-04-13T03:32:41.737546+0000","last_peered":"2024-04-13T03:32:41.737546+0000","last_clean":"2024-04-13T03:32:41.737546+0000","last_became_active":"2024-04-13T03:31:41.830246+0000","last_became_peered":"2024-04-13T03:31:41.830246+0000","last_unstale":"2024-04-13T03:32:41.737546+0000","last_undegraded":"2024-04-13T03:32:41.737546+0000","last_fullsized":"2024-04-13T03:32:41.737546+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:08:16.624658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.733886+0000","last_change":"2024-04-13T03:31:36.585534+0000","last_active":"2024-04-13T03:32:41.733886+0000","last_peered":"2024-04-13T03:32:41.733886+0000","last_clean":"2024-04-13T03:32:41.733886+0000","last_became_active":"2024-04-13T03:31:36.585324+0000","last_became_peered":"2024-04-13T03:31:36.585324+0000","last_unstale":"2024-04-13T03:32:41.733886+0000","last_undegraded":"2024-04-13T03:32:41.733886+0000","last_fullsized":"2024-04-13T03:32:41.733886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:12:35.971994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.885000+0000","last_change":"2024-04-13T03:31:37.587440+0000","last_active":"2024-04-13T03:33:09.885000+0000","last_peered":"2024-04-13T03:33:09.885000+0000","last_clean":"2024-04-13T03:33:09.885000+0000","last_became_active":"2024-04-13T03:31:37.581413+0000","last_became_peered":"2024-04-13T03:31:37.581413+0000","last_unstale":"2024-04-13T03:33:09.885000+0000","last_undegraded":"2024-04-13T03:33:09.885000+0000","last_fullsized":"2024-04-13T03:33:09.885000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:14:26.066107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731161+0000","last_change":"2024-04-13T03:31:41.848451+0000","last_active":"2024-04-13T03:32:41.731161+0000","last_peered":"2024-04-13T03:32:41.731161+0000","last_clean":"2024-04-13T03:32:41.731161+0000","last_became_active":"2024-04-13T03:31:41.848019+0000","last_became_peered":"2024-04-13T03:31:41.848019+0000","last_unstale":"2024-04-13T03:32:41.731161+0000","last_undegraded":"2024-04-13T03:32:41.731161+0000","last_fullsized":"2024-04-13T03:32:41.731161+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:26:34.558142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741735+0000","last_change":"2024-04-13T03:31:36.588393+0000","last_active":"2024-04-13T03:32:41.741735+0000","last_peered":"2024-04-13T03:32:41.741735+0000","last_clean":"2024-04-13T03:32:41.741735+0000","last_became_active":"2024-04-13T03:31:36.587184+0000","last_became_peered":"2024-04-13T03:31:36.587184+0000","last_unstale":"2024-04-13T03:32:41.741735+0000","last_undegraded":"2024-04-13T03:32:41.741735+0000","last_fullsized":"2024-04-13T03:32:41.741735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:44:24.985634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.892252+0000","last_change":"2024-04-13T03:31:37.600417+0000","last_active":"2024-04-13T03:33:09.892252+0000","last_peered":"2024-04-13T03:33:09.892252+0000","last_clean":"2024-04-13T03:33:09.892252+0000","last_became_active":"2024-04-13T03:31:37.599889+0000","last_became_peered":"2024-04-13T03:31:37.599889+0000","last_unstale":"2024-04-13T03:33:09.892252+0000","last_undegraded":"2024-04-13T03:33:09.892252+0000","last_fullsized":"2024-04-13T03:33:09.892252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:19:13.039127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.735636+0000","last_change":"2024-04-13T03:31:41.848432+0000","last_active":"2024-04-13T03:32:41.735636+0000","last_peered":"2024-04-13T03:32:41.735636+0000","last_clean":"2024-04-13T03:32:41.735636+0000","last_became_active":"2024-04-13T03:31:41.843515+0000","last_became_peered":"2024-04-13T03:31:41.843515+0000","last_unstale":"2024-04-13T03:32:41.735636+0000","last_undegraded":"2024-04-13T03:32:41.735636+0000","last_fullsized":"2024-04-13T03:32:41.735636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:31:15.512541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.733101+0000","last_change":"2024-04-13T03:31:36.560090+0000","last_active":"2024-04-13T03:32:41.733101+0000","last_peered":"2024-04-13T03:32:41.733101+0000","last_clean":"2024-04-13T03:32:41.733101+0000","last_became_active":"2024-04-13T03:31:36.559770+0000","last_became_peered":"2024-04-13T03:31:36.559770+0000","last_unstale":"2024-04-13T03:32:41.733101+0000","last_undegraded":"2024-04-13T03:32:41.733101+0000","last_fullsized":"2024-04-13T03:32:41.733101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:45:51.715456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.888658+0000","last_change":"2024-04-13T03:31:37.593845+0000","last_active":"2024-04-13T03:33:09.888658+0000","last_peered":"2024-04-13T03:33:09.888658+0000","last_clean":"2024-04-13T03:33:09.888658+0000","last_became_active":"2024-04-13T03:31:37.593484+0000","last_became_peered":"2024-04-13T03:31:37.593484+0000","last_unstale":"2024-04-13T03:33:09.888658+0000","last_undegraded":"2024-04-13T03:33:09.888658+0000","last_fullsized":"2024-04-13T03:33:09.888658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:38:05.817746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.908619+0000","last_change":"2024-04-13T03:31:41.830949+0000","last_active":"2024-04-13T03:32:41.908619+0000","last_peered":"2024-04-13T03:32:41.908619+0000","last_clean":"2024-04-13T03:32:41.908619+0000","last_became_active":"2024-04-13T03:31:41.830540+0000","last_became_peered":"2024-04-13T03:31:41.830540+0000","last_unstale":"2024-04-13T03:32:41.908619+0000","last_undegraded":"2024-04-13T03:32:41.908619+0000","last_fullsized":"2024-04-13T03:32:41.908619+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:09:00.006158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'5","reported_seq":196,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:28.927596+0000","last_change":"2024-04-13T03:31:36.574773+0000","last_active":"2024-04-13T03:33:28.927596+0000","last_peered":"2024-04-13T03:33:28.927596+0000","last_clean":"2024-04-13T03:33:28.927596+0000","last_became_active":"2024-04-13T03:31:36.574511+0000","last_became_peered":"2024-04-13T03:31:36.574511+0000","last_unstale":"2024-04-13T03:33:28.927596+0000","last_undegraded":"2024-04-13T03:33:28.927596+0000","last_fullsized":"2024-04-13T03:33:28.927596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:26:41.010269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":164,"num_read_kb":164,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.883268+0000","last_change":"2024-04-13T03:31:37.583383+0000","last_active":"2024-04-13T03:33:09.883268+0000","last_peered":"2024-04-13T03:33:09.883268+0000","last_clean":"2024-04-13T03:33:09.883268+0000","last_became_active":"2024-04-13T03:31:37.583026+0000","last_became_peered":"2024-04-13T03:31:37.583026+0000","last_unstale":"2024-04-13T03:33:09.883268+0000","last_undegraded":"2024-04-13T03:33:09.883268+0000","last_fullsized":"2024-04-13T03:33:09.883268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:34.790848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728631+0000","last_change":"2024-04-13T03:31:41.845867+0000","last_active":"2024-04-13T03:32:41.728631+0000","last_peered":"2024-04-13T03:32:41.728631+0000","last_clean":"2024-04-13T03:32:41.728631+0000","last_became_active":"2024-04-13T03:31:41.845408+0000","last_became_peered":"2024-04-13T03:31:41.845408+0000","last_unstale":"2024-04-13T03:32:41.728631+0000","last_undegraded":"2024-04-13T03:32:41.728631+0000","last_fullsized":"2024-04-13T03:32:41.728631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:10:42.830683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.730799+0000","last_change":"2024-04-13T03:31:36.562534+0000","last_active":"2024-04-13T03:32:41.730799+0000","last_peered":"2024-04-13T03:32:41.730799+0000","last_clean":"2024-04-13T03:32:41.730799+0000","last_became_active":"2024-04-13T03:31:36.562189+0000","last_became_peered":"2024-04-13T03:31:36.562189+0000","last_unstale":"2024-04-13T03:32:41.730799+0000","last_undegraded":"2024-04-13T03:32:41.730799+0000","last_fullsized":"2024-04-13T03:32:41.730799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:41:27.504982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.890232+0000","last_change":"2024-04-13T03:31:37.588637+0000","last_active":"2024-04-13T03:33:09.890232+0000","last_peered":"2024-04-13T03:33:09.890232+0000","last_clean":"2024-04-13T03:33:09.890232+0000","last_became_active":"2024-04-13T03:31:37.583263+0000","last_became_peered":"2024-04-13T03:31:37.583263+0000","last_unstale":"2024-04-13T03:33:09.890232+0000","last_undegraded":"2024-04-13T03:33:09.890232+0000","last_fullsized":"2024-04-13T03:33:09.890232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:07:40.503310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728659+0000","last_change":"2024-04-13T03:31:41.827302+0000","last_active":"2024-04-13T03:32:41.728659+0000","last_peered":"2024-04-13T03:32:41.728659+0000","last_clean":"2024-04-13T03:32:41.728659+0000","last_became_active":"2024-04-13T03:31:41.826822+0000","last_became_peered":"2024-04-13T03:31:41.826822+0000","last_unstale":"2024-04-13T03:32:41.728659+0000","last_undegraded":"2024-04-13T03:32:41.728659+0000","last_fullsized":"2024-04-13T03:32:41.728659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:00:16.589319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732293+0000","last_change":"2024-04-13T03:31:36.582710+0000","last_active":"2024-04-13T03:32:41.732293+0000","last_peered":"2024-04-13T03:32:41.732293+0000","last_clean":"2024-04-13T03:32:41.732293+0000","last_became_active":"2024-04-13T03:31:36.582372+0000","last_became_peered":"2024-04-13T03:31:36.582372+0000","last_unstale":"2024-04-13T03:32:41.732293+0000","last_undegraded":"2024-04-13T03:32:41.732293+0000","last_fullsized":"2024-04-13T03:32:41.732293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:38:58.990256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.886658+0000","last_change":"2024-04-13T03:31:37.578345+0000","last_active":"2024-04-13T03:33:09.886658+0000","last_peered":"2024-04-13T03:33:09.886658+0000","last_clean":"2024-04-13T03:33:09.886658+0000","last_became_active":"2024-04-13T03:31:37.578026+0000","last_became_peered":"2024-04-13T03:31:37.578026+0000","last_unstale":"2024-04-13T03:33:09.886658+0000","last_undegraded":"2024-04-13T03:33:09.886658+0000","last_fullsized":"2024-04-13T03:33:09.886658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:59:02.080243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.729113+0000","last_change":"2024-04-13T03:31:41.829108+0000","last_active":"2024-04-13T03:32:41.729113+0000","last_peered":"2024-04-13T03:32:41.729113+0000","last_clean":"2024-04-13T03:32:41.729113+0000","last_became_active":"2024-04-13T03:31:41.828603+0000","last_became_peered":"2024-04-13T03:31:41.828603+0000","last_unstale":"2024-04-13T03:32:41.729113+0000","last_undegraded":"2024-04-13T03:32:41.729113+0000","last_fullsized":"2024-04-13T03:32:41.729113+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:06:32.753928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'38","reported_seq":255,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:28.817713+0000","last_change":"2024-04-13T03:32:41.746579+0000","last_active":"2024-04-13T03:33:28.817713+0000","last_peered":"2024-04-13T03:33:28.817713+0000","last_clean":"2024-04-13T03:33:28.817713+0000","last_became_active":"2024-04-13T03:32:41.737639+0000","last_became_peered":"2024-04-13T03:32:41.737639+0000","last_unstale":"2024-04-13T03:33:28.817713+0000","last_undegraded":"2024-04-13T03:33:28.817713+0000","last_fullsized":"2024-04-13T03:33:28.817713+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:15:09.694242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":170,"num_read_kb":148,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":64206,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.860182+0000","last_change":"2024-04-13T03:31:37.580515+0000","last_active":"2024-04-13T03:33:09.860182+0000","last_peered":"2024-04-13T03:33:09.860182+0000","last_clean":"2024-04-13T03:33:09.860182+0000","last_became_active":"2024-04-13T03:31:37.580157+0000","last_became_peered":"2024-04-13T03:31:37.580157+0000","last_unstale":"2024-04-13T03:33:09.860182+0000","last_undegraded":"2024-04-13T03:33:09.860182+0000","last_fullsized":"2024-04-13T03:33:09.860182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:17:17.843273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732587+0000","last_change":"2024-04-13T03:31:41.833555+0000","last_active":"2024-04-13T03:32:41.732587+0000","last_peered":"2024-04-13T03:32:41.732587+0000","last_clean":"2024-04-13T03:32:41.732587+0000","last_became_active":"2024-04-13T03:31:41.833134+0000","last_became_peered":"2024-04-13T03:31:41.833134+0000","last_unstale":"2024-04-13T03:32:41.732587+0000","last_undegraded":"2024-04-13T03:32:41.732587+0000","last_fullsized":"2024-04-13T03:32:41.732587+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:22:58.311184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.737030+0000","last_change":"2024-04-13T03:31:36.591127+0000","last_active":"2024-04-13T03:32:41.737030+0000","last_peered":"2024-04-13T03:32:41.737030+0000","last_clean":"2024-04-13T03:32:41.737030+0000","last_became_active":"2024-04-13T03:31:36.590819+0000","last_became_peered":"2024-04-13T03:31:36.590819+0000","last_unstale":"2024-04-13T03:32:41.737030+0000","last_undegraded":"2024-04-13T03:32:41.737030+0000","last_fullsized":"2024-04-13T03:32:41.737030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:54:51.637340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.888324+0000","last_change":"2024-04-13T03:32:41.744284+0000","last_active":"2024-04-13T03:33:09.888324+0000","last_peered":"2024-04-13T03:33:09.888324+0000","last_clean":"2024-04-13T03:33:09.888324+0000","last_became_active":"2024-04-13T03:32:41.743848+0000","last_became_peered":"2024-04-13T03:32:41.743848+0000","last_unstale":"2024-04-13T03:33:09.888324+0000","last_undegraded":"2024-04-13T03:33:09.888324+0000","last_fullsized":"2024-04-13T03:33:09.888324+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:11:23.967329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731075+0000","last_change":"2024-04-13T03:31:41.833559+0000","last_active":"2024-04-13T03:32:41.731075+0000","last_peered":"2024-04-13T03:32:41.731075+0000","last_clean":"2024-04-13T03:32:41.731075+0000","last_became_active":"2024-04-13T03:31:41.833067+0000","last_became_peered":"2024-04-13T03:31:41.833067+0000","last_unstale":"2024-04-13T03:32:41.731075+0000","last_undegraded":"2024-04-13T03:32:41.731075+0000","last_fullsized":"2024-04-13T03:32:41.731075+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:56:09.027293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732615+0000","last_change":"2024-04-13T03:31:36.597136+0000","last_active":"2024-04-13T03:32:41.732615+0000","last_peered":"2024-04-13T03:32:41.732615+0000","last_clean":"2024-04-13T03:32:41.732615+0000","last_became_active":"2024-04-13T03:31:36.596789+0000","last_became_peered":"2024-04-13T03:31:36.596789+0000","last_unstale":"2024-04-13T03:32:41.732615+0000","last_undegraded":"2024-04-13T03:32:41.732615+0000","last_fullsized":"2024-04-13T03:32:41.732615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:39:28.371669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.893295+0000","last_change":"2024-04-13T03:31:37.584424+0000","last_active":"2024-04-13T03:33:09.893295+0000","last_peered":"2024-04-13T03:33:09.893295+0000","last_clean":"2024-04-13T03:33:09.893295+0000","last_became_active":"2024-04-13T03:31:37.584125+0000","last_became_peered":"2024-04-13T03:31:37.584125+0000","last_unstale":"2024-04-13T03:33:09.893295+0000","last_undegraded":"2024-04-13T03:33:09.893295+0000","last_fullsized":"2024-04-13T03:33:09.893295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:07:53.625486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731385+0000","last_change":"2024-04-13T03:31:41.852755+0000","last_active":"2024-04-13T03:32:41.731385+0000","last_peered":"2024-04-13T03:32:41.731385+0000","last_clean":"2024-04-13T03:32:41.731385+0000","last_became_active":"2024-04-13T03:31:41.851975+0000","last_became_peered":"2024-04-13T03:31:41.851975+0000","last_unstale":"2024-04-13T03:32:41.731385+0000","last_undegraded":"2024-04-13T03:32:41.731385+0000","last_fullsized":"2024-04-13T03:32:41.731385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:19:40.982525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"92'7","reported_seq":121,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:29.999182+0000","last_change":"2024-04-13T03:31:36.559351+0000","last_active":"2024-04-13T03:33:29.999182+0000","last_peered":"2024-04-13T03:33:29.999182+0000","last_clean":"2024-04-13T03:33:29.999182+0000","last_became_active":"2024-04-13T03:31:36.559011+0000","last_became_peered":"2024-04-13T03:31:36.559011+0000","last_unstale":"2024-04-13T03:33:29.999182+0000","last_undegraded":"2024-04-13T03:33:29.999182+0000","last_fullsized":"2024-04-13T03:33:29.999182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:08:08.983088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.878079+0000","last_change":"2024-04-13T03:31:37.580884+0000","last_active":"2024-04-13T03:33:09.878079+0000","last_peered":"2024-04-13T03:33:09.878079+0000","last_clean":"2024-04-13T03:33:09.878079+0000","last_became_active":"2024-04-13T03:31:37.580568+0000","last_became_peered":"2024-04-13T03:31:37.580568+0000","last_unstale":"2024-04-13T03:33:09.878079+0000","last_undegraded":"2024-04-13T03:33:09.878079+0000","last_fullsized":"2024-04-13T03:33:09.878079+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:42:41.484614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741856+0000","last_change":"2024-04-13T03:31:41.840912+0000","last_active":"2024-04-13T03:32:41.741856+0000","last_peered":"2024-04-13T03:32:41.741856+0000","last_clean":"2024-04-13T03:32:41.741856+0000","last_became_active":"2024-04-13T03:31:41.840129+0000","last_became_peered":"2024-04-13T03:31:41.840129+0000","last_unstale":"2024-04-13T03:32:41.741856+0000","last_undegraded":"2024-04-13T03:32:41.741856+0000","last_fullsized":"2024-04-13T03:32:41.741856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:45:33.716345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728411+0000","last_change":"2024-04-13T03:31:36.580230+0000","last_active":"2024-04-13T03:32:41.728411+0000","last_peered":"2024-04-13T03:32:41.728411+0000","last_clean":"2024-04-13T03:32:41.728411+0000","last_became_active":"2024-04-13T03:31:36.579874+0000","last_became_peered":"2024-04-13T03:31:36.579874+0000","last_unstale":"2024-04-13T03:32:41.728411+0000","last_undegraded":"2024-04-13T03:32:41.728411+0000","last_fullsized":"2024-04-13T03:32:41.728411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:30:17.154748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.882696+0000","last_change":"2024-04-13T03:31:37.581937+0000","last_active":"2024-04-13T03:33:09.882696+0000","last_peered":"2024-04-13T03:33:09.882696+0000","last_clean":"2024-04-13T03:33:09.882696+0000","last_became_active":"2024-04-13T03:31:37.581547+0000","last_became_peered":"2024-04-13T03:31:37.581547+0000","last_unstale":"2024-04-13T03:33:09.882696+0000","last_undegraded":"2024-04-13T03:33:09.882696+0000","last_fullsized":"2024-04-13T03:33:09.882696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:19:28.600996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732541+0000","last_change":"2024-04-13T03:31:41.842887+0000","last_active":"2024-04-13T03:32:41.732541+0000","last_peered":"2024-04-13T03:32:41.732541+0000","last_clean":"2024-04-13T03:32:41.732541+0000","last_became_active":"2024-04-13T03:31:41.842537+0000","last_became_peered":"2024-04-13T03:31:41.842537+0000","last_unstale":"2024-04-13T03:32:41.732541+0000","last_undegraded":"2024-04-13T03:32:41.732541+0000","last_fullsized":"2024-04-13T03:32:41.732541+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:44:38.930759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.729018+0000","last_change":"2024-04-13T03:31:36.561347+0000","last_active":"2024-04-13T03:32:41.729018+0000","last_peered":"2024-04-13T03:32:41.729018+0000","last_clean":"2024-04-13T03:32:41.729018+0000","last_became_active":"2024-04-13T03:31:36.560969+0000","last_became_peered":"2024-04-13T03:31:36.560969+0000","last_unstale":"2024-04-13T03:32:41.729018+0000","last_undegraded":"2024-04-13T03:32:41.729018+0000","last_fullsized":"2024-04-13T03:32:41.729018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:59:38.825930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.869287+0000","last_change":"2024-04-13T03:31:37.587599+0000","last_active":"2024-04-13T03:33:09.869287+0000","last_peered":"2024-04-13T03:33:09.869287+0000","last_clean":"2024-04-13T03:33:09.869287+0000","last_became_active":"2024-04-13T03:31:37.587257+0000","last_became_peered":"2024-04-13T03:31:37.587257+0000","last_unstale":"2024-04-13T03:33:09.869287+0000","last_undegraded":"2024-04-13T03:33:09.869287+0000","last_fullsized":"2024-04-13T03:33:09.869287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:39:57.956174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728661+0000","last_change":"2024-04-13T03:31:41.831134+0000","last_active":"2024-04-13T03:32:41.728661+0000","last_peered":"2024-04-13T03:32:41.728661+0000","last_clean":"2024-04-13T03:32:41.728661+0000","last_became_active":"2024-04-13T03:31:41.830671+0000","last_became_peered":"2024-04-13T03:31:41.830671+0000","last_unstale":"2024-04-13T03:32:41.728661+0000","last_undegraded":"2024-04-13T03:32:41.728661+0000","last_fullsized":"2024-04-13T03:32:41.728661+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:50:59.527951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731300+0000","last_change":"2024-04-13T03:31:36.586008+0000","last_active":"2024-04-13T03:32:41.731300+0000","last_peered":"2024-04-13T03:32:41.731300+0000","last_clean":"2024-04-13T03:32:41.731300+0000","last_became_active":"2024-04-13T03:31:36.583937+0000","last_became_peered":"2024-04-13T03:31:36.583937+0000","last_unstale":"2024-04-13T03:32:41.731300+0000","last_undegraded":"2024-04-13T03:32:41.731300+0000","last_fullsized":"2024-04-13T03:32:41.731300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:17:06.194397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.888950+0000","last_change":"2024-04-13T03:31:37.586185+0000","last_active":"2024-04-13T03:33:09.888950+0000","last_peered":"2024-04-13T03:33:09.888950+0000","last_clean":"2024-04-13T03:33:09.888950+0000","last_became_active":"2024-04-13T03:31:37.585509+0000","last_became_peered":"2024-04-13T03:31:37.585509+0000","last_unstale":"2024-04-13T03:33:09.888950+0000","last_undegraded":"2024-04-13T03:33:09.888950+0000","last_fullsized":"2024-04-13T03:33:09.888950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:35:41.064317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.740694+0000","last_change":"2024-04-13T03:31:41.847376+0000","last_active":"2024-04-13T03:32:41.740694+0000","last_peered":"2024-04-13T03:32:41.740694+0000","last_clean":"2024-04-13T03:32:41.740694+0000","last_became_active":"2024-04-13T03:31:41.846724+0000","last_became_peered":"2024-04-13T03:31:41.846724+0000","last_unstale":"2024-04-13T03:32:41.740694+0000","last_undegraded":"2024-04-13T03:32:41.740694+0000","last_fullsized":"2024-04-13T03:32:41.740694+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:26:14.046875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":117,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:29.982551+0000","last_change":"2024-04-13T03:31:36.582776+0000","last_active":"2024-04-13T03:33:29.982551+0000","last_peered":"2024-04-13T03:33:29.982551+0000","last_clean":"2024-04-13T03:33:29.982551+0000","last_became_active":"2024-04-13T03:31:36.582405+0000","last_became_peered":"2024-04-13T03:31:36.582405+0000","last_unstale":"2024-04-13T03:33:29.982551+0000","last_undegraded":"2024-04-13T03:33:29.982551+0000","last_fullsized":"2024-04-13T03:33:29.982551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:18:56.624442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.866497+0000","last_change":"2024-04-13T03:31:37.580552+0000","last_active":"2024-04-13T03:33:09.866497+0000","last_peered":"2024-04-13T03:33:09.866497+0000","last_clean":"2024-04-13T03:33:09.866497+0000","last_became_active":"2024-04-13T03:31:37.576390+0000","last_became_peered":"2024-04-13T03:31:37.576390+0000","last_unstale":"2024-04-13T03:33:09.866497+0000","last_undegraded":"2024-04-13T03:33:09.866497+0000","last_fullsized":"2024-04-13T03:33:09.866497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:50:48.328048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741384+0000","last_change":"2024-04-13T03:31:41.849818+0000","last_active":"2024-04-13T03:32:41.741384+0000","last_peered":"2024-04-13T03:32:41.741384+0000","last_clean":"2024-04-13T03:32:41.741384+0000","last_became_active":"2024-04-13T03:31:41.848846+0000","last_became_peered":"2024-04-13T03:31:41.848846+0000","last_unstale":"2024-04-13T03:32:41.741384+0000","last_undegraded":"2024-04-13T03:32:41.741384+0000","last_fullsized":"2024-04-13T03:32:41.741384+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:55:16.764935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.868852+0000","last_change":"2024-04-13T03:31:37.581723+0000","last_active":"2024-04-13T03:33:09.868852+0000","last_peered":"2024-04-13T03:33:09.868852+0000","last_clean":"2024-04-13T03:33:09.868852+0000","last_became_active":"2024-04-13T03:31:37.578857+0000","last_became_peered":"2024-04-13T03:31:37.578857+0000","last_unstale":"2024-04-13T03:33:09.868852+0000","last_undegraded":"2024-04-13T03:33:09.868852+0000","last_fullsized":"2024-04-13T03:33:09.868852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:47:34.447256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'6","reported_seq":120,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:32.003033+0000","last_change":"2024-04-13T03:31:36.592275+0000","last_active":"2024-04-13T03:33:32.003033+0000","last_peered":"2024-04-13T03:33:32.003033+0000","last_clean":"2024-04-13T03:33:32.003033+0000","last_became_active":"2024-04-13T03:31:36.591905+0000","last_became_peered":"2024-04-13T03:31:36.591905+0000","last_unstale":"2024-04-13T03:33:32.003033+0000","last_undegraded":"2024-04-13T03:33:32.003033+0000","last_fullsized":"2024-04-13T03:33:32.003033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.730575+0000","last_change":"2024-04-13T03:31:41.835412+0000","last_active":"2024-04-13T03:32:41.730575+0000","last_peered":"2024-04-13T03:32:41.730575+0000","last_clean":"2024-04-13T03:32:41.730575+0000","last_became_active":"2024-04-13T03:31:41.834978+0000","last_became_peered":"2024-04-13T03:31:41.834978+0000","last_unstale":"2024-04-13T03:32:41.730575+0000","last_undegraded":"2024-04-13T03:32:41.730575+0000","last_fullsized":"2024-04-13T03:32:41.730575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:45:52.037452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887114+0000","last_change":"2024-04-13T03:32:41.736526+0000","last_active":"2024-04-13T03:33:09.887114+0000","last_peered":"2024-04-13T03:33:09.887114+0000","last_clean":"2024-04-13T03:33:09.887114+0000","last_became_active":"2024-04-13T03:32:41.736021+0000","last_became_peered":"2024-04-13T03:32:41.736021+0000","last_unstale":"2024-04-13T03:33:09.887114+0000","last_undegraded":"2024-04-13T03:33:09.887114+0000","last_fullsized":"2024-04-13T03:33:09.887114+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:54:13.855368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728528+0000","last_change":"2024-04-13T03:31:36.580058+0000","last_active":"2024-04-13T03:32:41.728528+0000","last_peered":"2024-04-13T03:32:41.728528+0000","last_clean":"2024-04-13T03:32:41.728528+0000","last_became_active":"2024-04-13T03:31:36.579777+0000","last_became_peered":"2024-04-13T03:31:36.579777+0000","last_unstale":"2024-04-13T03:32:41.728528+0000","last_undegraded":"2024-04-13T03:32:41.728528+0000","last_fullsized":"2024-04-13T03:32:41.728528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:27:50.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":362,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.730653+0000","last_change":"2024-04-13T03:30:17.781014+0000","last_active":"2024-04-13T03:32:41.730653+0000","last_peered":"2024-04-13T03:32:41.730653+0000","last_clean":"2024-04-13T03:32:41.730653+0000","last_became_active":"2024-04-13T03:30:17.780417+0000","last_became_peered":"2024-04-13T03:30:17.780417+0000","last_unstale":"2024-04-13T03:32:41.730653+0000","last_undegraded":"2024-04-13T03:32:41.730653+0000","last_fullsized":"2024-04-13T03:32:41.730653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:26:22.174408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:26:22.174408+0000","last_clean_scrub_stamp":"2024-04-13T03:26:22.174408+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:52:50.418399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732734+0000","last_change":"2024-04-13T03:31:41.837973+0000","last_active":"2024-04-13T03:32:41.732734+0000","last_peered":"2024-04-13T03:32:41.732734+0000","last_clean":"2024-04-13T03:32:41.732734+0000","last_became_active":"2024-04-13T03:31:41.837562+0000","last_became_peered":"2024-04-13T03:31:41.837562+0000","last_unstale":"2024-04-13T03:32:41.732734+0000","last_undegraded":"2024-04-13T03:32:41.732734+0000","last_fullsized":"2024-04-13T03:32:41.732734+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:05:09.900193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.884631+0000","last_change":"2024-04-13T03:31:37.584279+0000","last_active":"2024-04-13T03:33:09.884631+0000","last_peered":"2024-04-13T03:33:09.884631+0000","last_clean":"2024-04-13T03:33:09.884631+0000","last_became_active":"2024-04-13T03:31:37.583891+0000","last_became_peered":"2024-04-13T03:31:37.583891+0000","last_unstale":"2024-04-13T03:33:09.884631+0000","last_undegraded":"2024-04-13T03:33:09.884631+0000","last_fullsized":"2024-04-13T03:33:09.884631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:56:21.446525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741100+0000","last_change":"2024-04-13T03:32:41.741100+0000","last_active":"2024-04-13T03:32:41.741100+0000","last_peered":"2024-04-13T03:32:41.741100+0000","last_clean":"2024-04-13T03:32:41.741100+0000","last_became_active":"2024-04-13T03:32:41.735297+0000","last_became_peered":"2024-04-13T03:32:41.735297+0000","last_unstale":"2024-04-13T03:32:41.741100+0000","last_undegraded":"2024-04-13T03:32:41.741100+0000","last_fullsized":"2024-04-13T03:32:41.741100+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:00:42.088105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731004+0000","last_change":"2024-04-13T03:31:41.833350+0000","last_active":"2024-04-13T03:32:41.731004+0000","last_peered":"2024-04-13T03:32:41.731004+0000","last_clean":"2024-04-13T03:32:41.731004+0000","last_became_active":"2024-04-13T03:31:41.832865+0000","last_became_peered":"2024-04-13T03:31:41.832865+0000","last_unstale":"2024-04-13T03:32:41.731004+0000","last_undegraded":"2024-04-13T03:32:41.731004+0000","last_fullsized":"2024-04-13T03:32:41.731004+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:23:31.872562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.751140+0000","last_change":"2024-04-13T03:32:41.750758+0000","last_active":"2024-04-13T03:32:41.751140+0000","last_peered":"2024-04-13T03:32:41.751140+0000","last_clean":"2024-04-13T03:32:41.751140+0000","last_became_active":"2024-04-13T03:32:41.740448+0000","last_became_peered":"2024-04-13T03:32:41.740448+0000","last_unstale":"2024-04-13T03:32:41.751140+0000","last_undegraded":"2024-04-13T03:32:41.751140+0000","last_fullsized":"2024-04-13T03:32:41.751140+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:07:30.563468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.884214+0000","last_change":"2024-04-13T03:31:37.584429+0000","last_active":"2024-04-13T03:33:09.884214+0000","last_peered":"2024-04-13T03:33:09.884214+0000","last_clean":"2024-04-13T03:33:09.884214+0000","last_became_active":"2024-04-13T03:31:37.583811+0000","last_became_peered":"2024-04-13T03:31:37.583811+0000","last_unstale":"2024-04-13T03:33:09.884214+0000","last_undegraded":"2024-04-13T03:33:09.884214+0000","last_fullsized":"2024-04-13T03:33:09.884214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:52:20.227001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.907911+0000","last_change":"2024-04-13T03:31:41.846065+0000","last_active":"2024-04-13T03:32:41.907911+0000","last_peered":"2024-04-13T03:32:41.907911+0000","last_clean":"2024-04-13T03:32:41.907911+0000","last_became_active":"2024-04-13T03:31:41.844422+0000","last_became_peered":"2024-04-13T03:31:41.844422+0000","last_unstale":"2024-04-13T03:32:41.907911+0000","last_undegraded":"2024-04-13T03:32:41.907911+0000","last_fullsized":"2024-04-13T03:32:41.907911+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:51:41.152066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'14","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731298+0000","last_change":"2024-04-13T03:31:36.598323+0000","last_active":"2024-04-13T03:32:41.731298+0000","last_peered":"2024-04-13T03:32:41.731298+0000","last_clean":"2024-04-13T03:32:41.731298+0000","last_became_active":"2024-04-13T03:31:36.597412+0000","last_became_peered":"2024-04-13T03:31:36.597412+0000","last_unstale":"2024-04-13T03:32:41.731298+0000","last_undegraded":"2024-04-13T03:32:41.731298+0000","last_fullsized":"2024-04-13T03:32:41.731298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:19:19.203273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.891938+0000","last_change":"2024-04-13T03:31:37.591599+0000","last_active":"2024-04-13T03:33:09.891938+0000","last_peered":"2024-04-13T03:33:09.891938+0000","last_clean":"2024-04-13T03:33:09.891938+0000","last_became_active":"2024-04-13T03:31:37.591243+0000","last_became_peered":"2024-04-13T03:31:37.591243+0000","last_unstale":"2024-04-13T03:33:09.891938+0000","last_undegraded":"2024-04-13T03:33:09.891938+0000","last_fullsized":"2024-04-13T03:33:09.891938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:11:35.460721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.736480+0000","last_change":"2024-04-13T03:31:41.821604+0000","last_active":"2024-04-13T03:32:41.736480+0000","last_peered":"2024-04-13T03:32:41.736480+0000","last_clean":"2024-04-13T03:32:41.736480+0000","last_became_active":"2024-04-13T03:31:41.820804+0000","last_became_peered":"2024-04-13T03:31:41.820804+0000","last_unstale":"2024-04-13T03:32:41.736480+0000","last_undegraded":"2024-04-13T03:32:41.736480+0000","last_fullsized":"2024-04-13T03:32:41.736480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:10:45.850099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728904+0000","last_change":"2024-04-13T03:31:36.582624+0000","last_active":"2024-04-13T03:32:41.728904+0000","last_peered":"2024-04-13T03:32:41.728904+0000","last_clean":"2024-04-13T03:32:41.728904+0000","last_became_active":"2024-04-13T03:31:36.582260+0000","last_became_peered":"2024-04-13T03:31:36.582260+0000","last_unstale":"2024-04-13T03:32:41.728904+0000","last_undegraded":"2024-04-13T03:32:41.728904+0000","last_fullsized":"2024-04-13T03:32:41.728904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:15.637016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.891518+0000","last_change":"2024-04-13T03:31:37.591286+0000","last_active":"2024-04-13T03:33:09.891518+0000","last_peered":"2024-04-13T03:33:09.891518+0000","last_clean":"2024-04-13T03:33:09.891518+0000","last_became_active":"2024-04-13T03:31:37.590891+0000","last_became_peered":"2024-04-13T03:31:37.590891+0000","last_unstale":"2024-04-13T03:33:09.891518+0000","last_undegraded":"2024-04-13T03:33:09.891518+0000","last_fullsized":"2024-04-13T03:33:09.891518+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:53:54.696082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728853+0000","last_change":"2024-04-13T03:31:41.830499+0000","last_active":"2024-04-13T03:32:41.728853+0000","last_peered":"2024-04-13T03:32:41.728853+0000","last_clean":"2024-04-13T03:32:41.728853+0000","last_became_active":"2024-04-13T03:31:41.830067+0000","last_became_peered":"2024-04-13T03:31:41.830067+0000","last_unstale":"2024-04-13T03:32:41.728853+0000","last_undegraded":"2024-04-13T03:32:41.728853+0000","last_fullsized":"2024-04-13T03:32:41.728853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:22.635885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.753157+0000","last_change":"2024-04-13T03:32:41.753157+0000","last_active":"2024-04-13T03:32:41.753157+0000","last_peered":"2024-04-13T03:32:41.753157+0000","last_clean":"2024-04-13T03:32:41.753157+0000","last_became_active":"2024-04-13T03:32:41.742206+0000","last_became_peered":"2024-04-13T03:32:41.742206+0000","last_unstale":"2024-04-13T03:32:41.753157+0000","last_undegraded":"2024-04-13T03:32:41.753157+0000","last_fullsized":"2024-04-13T03:32:41.753157+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:27:46.799187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.846009+0000","last_change":"2024-04-13T03:31:37.575749+0000","last_active":"2024-04-13T03:33:09.846009+0000","last_peered":"2024-04-13T03:33:09.846009+0000","last_clean":"2024-04-13T03:33:09.846009+0000","last_became_active":"2024-04-13T03:31:37.574152+0000","last_became_peered":"2024-04-13T03:31:37.574152+0000","last_unstale":"2024-04-13T03:33:09.846009+0000","last_undegraded":"2024-04-13T03:33:09.846009+0000","last_fullsized":"2024-04-13T03:33:09.846009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:38:15.710978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731574+0000","last_change":"2024-04-13T03:31:41.836777+0000","last_active":"2024-04-13T03:32:41.731574+0000","last_peered":"2024-04-13T03:32:41.731574+0000","last_clean":"2024-04-13T03:32:41.731574+0000","last_became_active":"2024-04-13T03:31:41.835877+0000","last_became_peered":"2024-04-13T03:31:41.835877+0000","last_unstale":"2024-04-13T03:32:41.731574+0000","last_undegraded":"2024-04-13T03:32:41.731574+0000","last_fullsized":"2024-04-13T03:32:41.731574+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:46:44.174095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.909200+0000","last_change":"2024-04-13T03:31:36.587981+0000","last_active":"2024-04-13T03:32:41.909200+0000","last_peered":"2024-04-13T03:32:41.909200+0000","last_clean":"2024-04-13T03:32:41.909200+0000","last_became_active":"2024-04-13T03:31:36.586013+0000","last_became_peered":"2024-04-13T03:31:36.586013+0000","last_unstale":"2024-04-13T03:32:41.909200+0000","last_undegraded":"2024-04-13T03:32:41.909200+0000","last_fullsized":"2024-04-13T03:32:41.909200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:16:27.075322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.872088+0000","last_change":"2024-04-13T03:31:37.594185+0000","last_active":"2024-04-13T03:33:09.872088+0000","last_peered":"2024-04-13T03:33:09.872088+0000","last_clean":"2024-04-13T03:33:09.872088+0000","last_became_active":"2024-04-13T03:31:37.593791+0000","last_became_peered":"2024-04-13T03:31:37.593791+0000","last_unstale":"2024-04-13T03:33:09.872088+0000","last_undegraded":"2024-04-13T03:33:09.872088+0000","last_fullsized":"2024-04-13T03:33:09.872088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:43:32.035371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728904+0000","last_change":"2024-04-13T03:31:41.830392+0000","last_active":"2024-04-13T03:32:41.728904+0000","last_peered":"2024-04-13T03:32:41.728904+0000","last_clean":"2024-04-13T03:32:41.728904+0000","last_became_active":"2024-04-13T03:31:41.829967+0000","last_became_peered":"2024-04-13T03:31:41.829967+0000","last_unstale":"2024-04-13T03:32:41.728904+0000","last_undegraded":"2024-04-13T03:32:41.728904+0000","last_fullsized":"2024-04-13T03:32:41.728904+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:43:25.326951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.736877+0000","last_change":"2024-04-13T03:31:36.597099+0000","last_active":"2024-04-13T03:32:41.736877+0000","last_peered":"2024-04-13T03:32:41.736877+0000","last_clean":"2024-04-13T03:32:41.736877+0000","last_became_active":"2024-04-13T03:31:36.596714+0000","last_became_peered":"2024-04-13T03:31:36.596714+0000","last_unstale":"2024-04-13T03:32:41.736877+0000","last_undegraded":"2024-04-13T03:32:41.736877+0000","last_fullsized":"2024-04-13T03:32:41.736877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:42:31.477730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.897485+0000","last_change":"2024-04-13T03:31:37.587038+0000","last_active":"2024-04-13T03:33:09.897485+0000","last_peered":"2024-04-13T03:33:09.897485+0000","last_clean":"2024-04-13T03:33:09.897485+0000","last_became_active":"2024-04-13T03:31:37.586618+0000","last_became_peered":"2024-04-13T03:31:37.586618+0000","last_unstale":"2024-04-13T03:33:09.897485+0000","last_undegraded":"2024-04-13T03:33:09.897485+0000","last_fullsized":"2024-04-13T03:33:09.897485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:57:33.868529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.729704+0000","last_change":"2024-04-13T03:31:41.833148+0000","last_active":"2024-04-13T03:32:41.729704+0000","last_peered":"2024-04-13T03:32:41.729704+0000","last_clean":"2024-04-13T03:32:41.729704+0000","last_became_active":"2024-04-13T03:31:41.832714+0000","last_became_peered":"2024-04-13T03:31:41.832714+0000","last_unstale":"2024-04-13T03:32:41.729704+0000","last_undegraded":"2024-04-13T03:32:41.729704+0000","last_fullsized":"2024-04-13T03:32:41.729704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:12:07.316488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728629+0000","last_change":"2024-04-13T03:31:36.564887+0000","last_active":"2024-04-13T03:32:41.728629+0000","last_peered":"2024-04-13T03:32:41.728629+0000","last_clean":"2024-04-13T03:32:41.728629+0000","last_became_active":"2024-04-13T03:31:36.564438+0000","last_became_peered":"2024-04-13T03:31:36.564438+0000","last_unstale":"2024-04-13T03:32:41.728629+0000","last_undegraded":"2024-04-13T03:32:41.728629+0000","last_fullsized":"2024-04-13T03:32:41.728629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:19.310322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.889488+0000","last_change":"2024-04-13T03:31:37.584358+0000","last_active":"2024-04-13T03:33:09.889488+0000","last_peered":"2024-04-13T03:33:09.889488+0000","last_clean":"2024-04-13T03:33:09.889488+0000","last_became_active":"2024-04-13T03:31:37.584005+0000","last_became_peered":"2024-04-13T03:31:37.584005+0000","last_unstale":"2024-04-13T03:33:09.889488+0000","last_undegraded":"2024-04-13T03:33:09.889488+0000","last_fullsized":"2024-04-13T03:33:09.889488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:07:21.465290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.733665+0000","last_change":"2024-04-13T03:31:41.833451+0000","last_active":"2024-04-13T03:32:41.733665+0000","last_peered":"2024-04-13T03:32:41.733665+0000","last_clean":"2024-04-13T03:32:41.733665+0000","last_became_active":"2024-04-13T03:31:41.832734+0000","last_became_peered":"2024-04-13T03:31:41.832734+0000","last_unstale":"2024-04-13T03:32:41.733665+0000","last_undegraded":"2024-04-13T03:32:41.733665+0000","last_fullsized":"2024-04-13T03:32:41.733665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:49:48.848152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":118,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:29.929988+0000","last_change":"2024-04-13T03:31:36.581697+0000","last_active":"2024-04-13T03:33:29.929988+0000","last_peered":"2024-04-13T03:33:29.929988+0000","last_clean":"2024-04-13T03:33:29.929988+0000","last_became_active":"2024-04-13T03:31:36.581345+0000","last_became_peered":"2024-04-13T03:31:36.581345+0000","last_unstale":"2024-04-13T03:33:29.929988+0000","last_undegraded":"2024-04-13T03:33:29.929988+0000","last_fullsized":"2024-04-13T03:33:29.929988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:29:53.192770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887198+0000","last_change":"2024-04-13T03:31:37.571879+0000","last_active":"2024-04-13T03:33:09.887198+0000","last_peered":"2024-04-13T03:33:09.887198+0000","last_clean":"2024-04-13T03:33:09.887198+0000","last_became_active":"2024-04-13T03:31:37.571618+0000","last_became_peered":"2024-04-13T03:31:37.571618+0000","last_unstale":"2024-04-13T03:33:09.887198+0000","last_undegraded":"2024-04-13T03:33:09.887198+0000","last_fullsized":"2024-04-13T03:33:09.887198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:42.920388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.734871+0000","last_change":"2024-04-13T03:31:41.851871+0000","last_active":"2024-04-13T03:32:41.734871+0000","last_peered":"2024-04-13T03:32:41.734871+0000","last_clean":"2024-04-13T03:32:41.734871+0000","last_became_active":"2024-04-13T03:31:41.851045+0000","last_became_peered":"2024-04-13T03:31:41.851045+0000","last_unstale":"2024-04-13T03:32:41.734871+0000","last_undegraded":"2024-04-13T03:32:41.734871+0000","last_fullsized":"2024-04-13T03:32:41.734871+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:35:41.937723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732348+0000","last_change":"2024-04-13T03:31:36.586110+0000","last_active":"2024-04-13T03:32:41.732348+0000","last_peered":"2024-04-13T03:32:41.732348+0000","last_clean":"2024-04-13T03:32:41.732348+0000","last_became_active":"2024-04-13T03:31:36.585832+0000","last_became_peered":"2024-04-13T03:31:36.585832+0000","last_unstale":"2024-04-13T03:32:41.732348+0000","last_undegraded":"2024-04-13T03:32:41.732348+0000","last_fullsized":"2024-04-13T03:32:41.732348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:00:57.825760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.884320+0000","last_change":"2024-04-13T03:31:37.580004+0000","last_active":"2024-04-13T03:33:09.884320+0000","last_peered":"2024-04-13T03:33:09.884320+0000","last_clean":"2024-04-13T03:33:09.884320+0000","last_became_active":"2024-04-13T03:31:37.579634+0000","last_became_peered":"2024-04-13T03:31:37.579634+0000","last_unstale":"2024-04-13T03:33:09.884320+0000","last_undegraded":"2024-04-13T03:33:09.884320+0000","last_fullsized":"2024-04-13T03:33:09.884320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:39:59.872588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.733211+0000","last_change":"2024-04-13T03:31:41.825456+0000","last_active":"2024-04-13T03:32:41.733211+0000","last_peered":"2024-04-13T03:32:41.733211+0000","last_clean":"2024-04-13T03:32:41.733211+0000","last_became_active":"2024-04-13T03:31:41.824929+0000","last_became_peered":"2024-04-13T03:31:41.824929+0000","last_unstale":"2024-04-13T03:32:41.733211+0000","last_undegraded":"2024-04-13T03:32:41.733211+0000","last_fullsized":"2024-04-13T03:32:41.733211+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:17:39.405147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741969+0000","last_change":"2024-04-13T03:31:36.595009+0000","last_active":"2024-04-13T03:32:41.741969+0000","last_peered":"2024-04-13T03:32:41.741969+0000","last_clean":"2024-04-13T03:32:41.741969+0000","last_became_active":"2024-04-13T03:31:36.594314+0000","last_became_peered":"2024-04-13T03:31:36.594314+0000","last_unstale":"2024-04-13T03:32:41.741969+0000","last_undegraded":"2024-04-13T03:32:41.741969+0000","last_fullsized":"2024-04-13T03:32:41.741969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:42:33.570088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.878633+0000","last_change":"2024-04-13T03:31:37.579899+0000","last_active":"2024-04-13T03:33:09.878633+0000","last_peered":"2024-04-13T03:33:09.878633+0000","last_clean":"2024-04-13T03:33:09.878633+0000","last_became_active":"2024-04-13T03:31:37.579345+0000","last_became_peered":"2024-04-13T03:31:37.579345+0000","last_unstale":"2024-04-13T03:33:09.878633+0000","last_undegraded":"2024-04-13T03:33:09.878633+0000","last_fullsized":"2024-04-13T03:33:09.878633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:06:55.646866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.868497+0000","last_change":"2024-04-13T03:31:37.586373+0000","last_active":"2024-04-13T03:33:09.868497+0000","last_peered":"2024-04-13T03:33:09.868497+0000","last_clean":"2024-04-13T03:33:09.868497+0000","last_became_active":"2024-04-13T03:31:37.586061+0000","last_became_peered":"2024-04-13T03:31:37.586061+0000","last_unstale":"2024-04-13T03:33:09.868497+0000","last_undegraded":"2024-04-13T03:33:09.868497+0000","last_fullsized":"2024-04-13T03:33:09.868497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:58:52.487187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.866958+0000","last_change":"2024-04-13T03:31:37.581375+0000","last_active":"2024-04-13T03:33:09.866958+0000","last_peered":"2024-04-13T03:33:09.866958+0000","last_clean":"2024-04-13T03:33:09.866958+0000","last_became_active":"2024-04-13T03:31:37.580802+0000","last_became_peered":"2024-04-13T03:31:37.580802+0000","last_unstale":"2024-04-13T03:33:09.866958+0000","last_undegraded":"2024-04-13T03:33:09.866958+0000","last_fullsized":"2024-04-13T03:33:09.866958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:41.895243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.892179+0000","last_change":"2024-04-13T03:31:37.580598+0000","last_active":"2024-04-13T03:33:09.892179+0000","last_peered":"2024-04-13T03:33:09.892179+0000","last_clean":"2024-04-13T03:33:09.892179+0000","last_became_active":"2024-04-13T03:31:37.580276+0000","last_became_peered":"2024-04-13T03:31:37.580276+0000","last_unstale":"2024-04-13T03:33:09.892179+0000","last_undegraded":"2024-04-13T03:33:09.892179+0000","last_fullsized":"2024-04-13T03:33:09.892179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:56:02.302300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.895650+0000","last_change":"2024-04-13T03:31:37.596435+0000","last_active":"2024-04-13T03:33:09.895650+0000","last_peered":"2024-04-13T03:33:09.895650+0000","last_clean":"2024-04-13T03:33:09.895650+0000","last_became_active":"2024-04-13T03:31:37.596095+0000","last_became_peered":"2024-04-13T03:31:37.596095+0000","last_unstale":"2024-04-13T03:33:09.895650+0000","last_undegraded":"2024-04-13T03:33:09.895650+0000","last_fullsized":"2024-04-13T03:33:09.895650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:04.425037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.895926+0000","last_change":"2024-04-13T03:31:37.578003+0000","last_active":"2024-04-13T03:33:09.895926+0000","last_peered":"2024-04-13T03:33:09.895926+0000","last_clean":"2024-04-13T03:33:09.895926+0000","last_became_active":"2024-04-13T03:31:37.577637+0000","last_became_peered":"2024-04-13T03:31:37.577637+0000","last_unstale":"2024-04-13T03:33:09.895926+0000","last_undegraded":"2024-04-13T03:33:09.895926+0000","last_fullsized":"2024-04-13T03:33:09.895926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:30:37.138611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.864965+0000","last_change":"2024-04-13T03:31:37.585404+0000","last_active":"2024-04-13T03:33:09.864965+0000","last_peered":"2024-04-13T03:33:09.864965+0000","last_clean":"2024-04-13T03:33:09.864965+0000","last_became_active":"2024-04-13T03:31:37.585060+0000","last_became_peered":"2024-04-13T03:31:37.585060+0000","last_unstale":"2024-04-13T03:33:09.864965+0000","last_undegraded":"2024-04-13T03:33:09.864965+0000","last_fullsized":"2024-04-13T03:33:09.864965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:37:52.165706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.865243+0000","last_change":"2024-04-13T03:31:37.576265+0000","last_active":"2024-04-13T03:33:09.865243+0000","last_peered":"2024-04-13T03:33:09.865243+0000","last_clean":"2024-04-13T03:33:09.865243+0000","last_became_active":"2024-04-13T03:31:37.575906+0000","last_became_peered":"2024-04-13T03:31:37.575906+0000","last_unstale":"2024-04-13T03:33:09.865243+0000","last_undegraded":"2024-04-13T03:33:09.865243+0000","last_fullsized":"2024-04-13T03:33:09.865243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:31:34.505830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.893424+0000","last_change":"2024-04-13T03:31:37.582865+0000","last_active":"2024-04-13T03:33:09.893424+0000","last_peered":"2024-04-13T03:33:09.893424+0000","last_clean":"2024-04-13T03:33:09.893424+0000","last_became_active":"2024-04-13T03:31:37.581055+0000","last_became_peered":"2024-04-13T03:31:37.581055+0000","last_unstale":"2024-04-13T03:33:09.893424+0000","last_undegraded":"2024-04-13T03:33:09.893424+0000","last_fullsized":"2024-04-13T03:33:09.893424+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:01:07.815483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.893615+0000","last_change":"2024-04-13T03:31:37.587677+0000","last_active":"2024-04-13T03:33:09.893615+0000","last_peered":"2024-04-13T03:33:09.893615+0000","last_clean":"2024-04-13T03:33:09.893615+0000","last_became_active":"2024-04-13T03:31:37.586778+0000","last_became_peered":"2024-04-13T03:31:37.586778+0000","last_unstale":"2024-04-13T03:33:09.893615+0000","last_undegraded":"2024-04-13T03:33:09.893615+0000","last_fullsized":"2024-04-13T03:33:09.893615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:52:13.968559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.894682+0000","last_change":"2024-04-13T03:31:37.577694+0000","last_active":"2024-04-13T03:33:09.894682+0000","last_peered":"2024-04-13T03:33:09.894682+0000","last_clean":"2024-04-13T03:33:09.894682+0000","last_became_active":"2024-04-13T03:31:37.577287+0000","last_became_peered":"2024-04-13T03:31:37.577287+0000","last_unstale":"2024-04-13T03:33:09.894682+0000","last_undegraded":"2024-04-13T03:33:09.894682+0000","last_fullsized":"2024-04-13T03:33:09.894682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:16:26.503330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887902+0000","last_change":"2024-04-13T03:31:37.591160+0000","last_active":"2024-04-13T03:33:09.887902+0000","last_peered":"2024-04-13T03:33:09.887902+0000","last_clean":"2024-04-13T03:33:09.887902+0000","last_became_active":"2024-04-13T03:31:37.590666+0000","last_became_peered":"2024-04-13T03:31:37.590666+0000","last_unstale":"2024-04-13T03:33:09.887902+0000","last_undegraded":"2024-04-13T03:33:09.887902+0000","last_fullsized":"2024-04-13T03:33:09.887902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:43:25.976108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.896823+0000","last_change":"2024-04-13T03:31:37.594433+0000","last_active":"2024-04-13T03:33:09.896823+0000","last_peered":"2024-04-13T03:33:09.896823+0000","last_clean":"2024-04-13T03:33:09.896823+0000","last_became_active":"2024-04-13T03:31:37.590942+0000","last_became_peered":"2024-04-13T03:31:37.590942+0000","last_unstale":"2024-04-13T03:33:09.896823+0000","last_undegraded":"2024-04-13T03:33:09.896823+0000","last_fullsized":"2024-04-13T03:33:09.896823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:42:47.534702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.869884+0000","last_change":"2024-04-13T03:31:37.583342+0000","last_active":"2024-04-13T03:33:09.869884+0000","last_peered":"2024-04-13T03:33:09.869884+0000","last_clean":"2024-04-13T03:33:09.869884+0000","last_became_active":"2024-04-13T03:31:37.582983+0000","last_became_peered":"2024-04-13T03:31:37.582983+0000","last_unstale":"2024-04-13T03:33:09.869884+0000","last_undegraded":"2024-04-13T03:33:09.869884+0000","last_fullsized":"2024-04-13T03:33:09.869884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:24:26.616843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284330,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71941,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":869,"num_read_kb":513,"num_write":253,"num_write_kb":149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":64318,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":215823,"data_compressed":14002,"data_compressed_allocated":20480,"data_compressed_original":77824,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":508208,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1159899,"data_compressed":58649,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":512122,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":508292,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1105574,"data_compressed":56822,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":503530,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":569015,"data_compressed":22492,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":504986,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":505312,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":503238,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":572591,"data_compressed":31013,"data_compressed_allocated":40960,"data_compressed_original":118784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":1100624,"data_compressed":56822,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66207,"data_compressed":1827,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":65687,"data_compressed":10348,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":66723,"data_compressed":1827,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T03:33:35.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph pg dump --format=json 2024-04-13T03:33:35.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:35 smithi119 ceph-mon[35835]: pgmap v379: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:35.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:35 smithi188 ceph-mon[36398]: pgmap v379: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:36.029 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:36.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:36 smithi016 ceph-mon[31772]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:33:36.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:36 smithi119 ceph-mon[35835]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:33:36.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:36 smithi188 ceph-mon[36398]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:33:37.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:37 smithi016 ceph-mon[31772]: pgmap v380: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:33:37.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:37 smithi119 ceph-mon[35835]: pgmap v380: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:33:37.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:37 smithi188 ceph-mon[36398]: pgmap v380: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:33:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.267+0000 7f8fc2200700 1 -- 172.21.15.16:0/2777816090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109cf0 msgr2=0x7f8fbc10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.267+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2777816090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109cf0 0x7f8fbc10dd40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80099c0 tx=0x7f8fa8009cd0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.267+0000 7f8fc2200700 1 -- 172.21.15.16:0/2777816090 shutdown_connections 2024-04-13T03:33:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.267+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2777816090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fbc10e420 0x7f8fbc115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.267+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2777816090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc109340 0x7f8fbc109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.267+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2777816090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109cf0 0x7f8fbc10dd40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.267+0000 7f8fc2200700 1 -- 172.21.15.16:0/2777816090 >> 172.21.15.16:0/2777816090 conn(0x7f8fbc076280 msgr2=0x7f8fbc1078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.268+0000 7f8fc2200700 1 -- 172.21.15.16:0/2777816090 shutdown_connections 2024-04-13T03:33:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.268+0000 7f8fc2200700 1 -- 172.21.15.16:0/2777816090 wait complete. 2024-04-13T03:33:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.269+0000 7f8fc2200700 1 Processor -- start 2024-04-13T03:33:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.269+0000 7f8fc2200700 1 -- start start 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fc2200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109340 0x7f8fbc1a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fc2200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc109cf0 0x7f8fbc1a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fc2200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fbc10e420 0x7f8fbc19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fc2200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc119430 con 0x7f8fbc109cf0 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fc2200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1192b0 con 0x7f8fbc10e420 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fc2200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc1195b0 con 0x7f8fbc109340 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fbb400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109340 0x7f8fbc1a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fbaa00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc109cf0 0x7f8fbc1a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fbb400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109340 0x7f8fbc1a5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:55958/0 (socket says 172.21.15.16:55958) 2024-04-13T03:33:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fbb400700 1 -- 172.21.15.16:0/2420094239 learned_addr learned my addr 172.21.15.16:0/2420094239 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:38.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.270+0000 7f8fbbe00700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fbc10e420 0x7f8fbc19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.271+0000 7f8fbb400700 1 -- 172.21.15.16:0/2420094239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fbc10e420 msgr2=0x7f8fbc19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.271+0000 7f8fbb400700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fbc10e420 0x7f8fbc19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.271+0000 7f8fbb400700 1 -- 172.21.15.16:0/2420094239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc109cf0 msgr2=0x7f8fbc1a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.271+0000 7f8fbb400700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc109cf0 0x7f8fbc1a5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.271+0000 7f8fbb400700 1 -- 172.21.15.16:0/2420094239 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc19ffd0 con 0x7f8fbc109340 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.271+0000 7f8fbbe00700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fbc10e420 0x7f8fbc19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.271+0000 7f8fbb400700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109340 0x7f8fbc1a5620 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00d950 tx=0x7f8fac00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.272+0000 7f8fb3e00700 1 -- 172.21.15.16:0/2420094239 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac016070 con 0x7f8fbc109340 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.272+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fbc1a02c0 con 0x7f8fbc109340 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.272+0000 7f8fb3e00700 1 -- 172.21.15.16:0/2420094239 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fac00fd70 con 0x7f8fbc109340 2024-04-13T03:33:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.272+0000 7f8fb3e00700 1 -- 172.21.15.16:0/2420094239 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac00e610 con 0x7f8fbc109340 2024-04-13T03:33:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.272+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fbc1ac5a0 con 0x7f8fbc109340 2024-04-13T03:33:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.273+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fbc10a820 con 0x7f8fbc109340 2024-04-13T03:33:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.275+0000 7f8fb3e00700 1 -- 172.21.15.16:0/2420094239 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8fac019070 con 0x7f8fbc109340 2024-04-13T03:33:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.277+0000 7f8fb3e00700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fa00a3c40 0x7f8fa00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.277+0000 7f8fb3e00700 1 -- 172.21.15.16:0/2420094239 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9092+0+0 (secure 0 0 0) 0x7f8fac0d0270 con 0x7f8fbc109340 2024-04-13T03:33:38.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.277+0000 7f8fbaa00700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fa00a3c40 0x7f8fa00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:38.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.278+0000 7f8fbaa00700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fa00a3c40 0x7f8fa00a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8000c00 tx=0x7f8fa8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:38.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.281+0000 7f8fb3e00700 1 -- 172.21.15.16:0/2420094239 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8fac09c890 con 0x7f8fbc109340 2024-04-13T03:33:38.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.516+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8fbc1a0c20 con 0x7f8fa00a3c40 2024-04-13T03:33:38.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.546+0000 7f8fb3e00700 1 -- 172.21.15.16:0/2420094239 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340846 (secure 0 0 0) 0x7f8fbc1a0c20 con 0x7f8fa00a3c40 2024-04-13T03:33:38.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:38.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.554+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fa00a3c40 msgr2=0x7f8fa00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.554+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fa00a3c40 0x7f8fa00a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8000c00 tx=0x7f8fa8005fb0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.554+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109340 msgr2=0x7f8fbc1a5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.554+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109340 0x7f8fbc1a5620 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00d950 tx=0x7f8fac00dc60 comp rx=0 tx=0).stop 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.555+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 shutdown_connections 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.555+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fa00a3c40 0x7f8fa00a6100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.555+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fbc10e420 0x7f8fbc19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.555+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8fbc109340 0x7f8fbc1a5620 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.555+0000 7f8fc2200700 1 --2- 172.21.15.16:0/2420094239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fbc109cf0 0x7f8fbc1a5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.555+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 >> 172.21.15.16:0/2420094239 conn(0x7f8fbc076280 msgr2=0x7f8fbc0768a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.555+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 shutdown_connections 2024-04-13T03:33:38.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:38.555+0000 7f8fc2200700 1 -- 172.21.15.16:0/2420094239 wait complete. 2024-04-13T03:33:38.557 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-13T03:33:39.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:39 smithi016 ceph-mon[31772]: pgmap v381: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:39.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:39 smithi016 ceph-mon[31772]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:33:39.488 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-04-13T03:33:37.922649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662543,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1050,"num_read_kb":652,"num_write":436,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1245054,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":859,"ondisk_log_size":859,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369868,"kb_used_data":12396,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571956,"statfs":{"total":1151940427776,"available":1151561682944,"internally_reserved":0,"allocated":12693504,"data_stored":8053391,"data_compressed":370453,"data_compressed_allocated":499712,"data_compressed_original":2535424,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006231"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.890185+0000","last_change":"2024-04-13T03:31:37.597340+0000","last_active":"2024-04-13T03:33:09.890185+0000","last_peered":"2024-04-13T03:33:09.890185+0000","last_clean":"2024-04-13T03:33:09.890185+0000","last_became_active":"2024-04-13T03:31:37.597081+0000","last_became_peered":"2024-04-13T03:31:37.597081+0000","last_unstale":"2024-04-13T03:33:09.890185+0000","last_undegraded":"2024-04-13T03:33:09.890185+0000","last_fullsized":"2024-04-13T03:33:09.890185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:10:36.679921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.875017+0000","last_change":"2024-04-13T03:31:37.578150+0000","last_active":"2024-04-13T03:33:09.875017+0000","last_peered":"2024-04-13T03:33:09.875017+0000","last_clean":"2024-04-13T03:33:09.875017+0000","last_became_active":"2024-04-13T03:31:37.574207+0000","last_became_peered":"2024-04-13T03:31:37.574207+0000","last_unstale":"2024-04-13T03:33:09.875017+0000","last_undegraded":"2024-04-13T03:33:09.875017+0000","last_fullsized":"2024-04-13T03:33:09.875017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:38:41.999046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.892570+0000","last_change":"2024-04-13T03:31:37.581782+0000","last_active":"2024-04-13T03:33:09.892570+0000","last_peered":"2024-04-13T03:33:09.892570+0000","last_clean":"2024-04-13T03:33:09.892570+0000","last_became_active":"2024-04-13T03:31:37.581448+0000","last_became_peered":"2024-04-13T03:31:37.581448+0000","last_unstale":"2024-04-13T03:33:09.892570+0000","last_undegraded":"2024-04-13T03:33:09.892570+0000","last_fullsized":"2024-04-13T03:33:09.892570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:56:44.538333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.864571+0000","last_change":"2024-04-13T03:31:37.578982+0000","last_active":"2024-04-13T03:33:09.864571+0000","last_peered":"2024-04-13T03:33:09.864571+0000","last_clean":"2024-04-13T03:33:09.864571+0000","last_became_active":"2024-04-13T03:31:37.578375+0000","last_became_peered":"2024-04-13T03:31:37.578375+0000","last_unstale":"2024-04-13T03:33:09.864571+0000","last_undegraded":"2024-04-13T03:33:09.864571+0000","last_fullsized":"2024-04-13T03:33:09.864571+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:13:36.312872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.884519+0000","last_change":"2024-04-13T03:31:37.577546+0000","last_active":"2024-04-13T03:33:09.884519+0000","last_peered":"2024-04-13T03:33:09.884519+0000","last_clean":"2024-04-13T03:33:09.884519+0000","last_became_active":"2024-04-13T03:31:37.577238+0000","last_became_peered":"2024-04-13T03:31:37.577238+0000","last_unstale":"2024-04-13T03:33:09.884519+0000","last_undegraded":"2024-04-13T03:33:09.884519+0000","last_fullsized":"2024-04-13T03:33:09.884519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:25:17.759986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.869740+0000","last_change":"2024-04-13T03:31:37.585806+0000","last_active":"2024-04-13T03:33:09.869740+0000","last_peered":"2024-04-13T03:33:09.869740+0000","last_clean":"2024-04-13T03:33:09.869740+0000","last_became_active":"2024-04-13T03:31:37.585448+0000","last_became_peered":"2024-04-13T03:31:37.585448+0000","last_unstale":"2024-04-13T03:33:09.869740+0000","last_undegraded":"2024-04-13T03:33:09.869740+0000","last_fullsized":"2024-04-13T03:33:09.869740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:32:19.336091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.878071+0000","last_change":"2024-04-13T03:32:41.738725+0000","last_active":"2024-04-13T03:33:09.878071+0000","last_peered":"2024-04-13T03:33:09.878071+0000","last_clean":"2024-04-13T03:33:09.878071+0000","last_became_active":"2024-04-13T03:32:41.738281+0000","last_became_peered":"2024-04-13T03:32:41.738281+0000","last_unstale":"2024-04-13T03:33:09.878071+0000","last_undegraded":"2024-04-13T03:33:09.878071+0000","last_fullsized":"2024-04-13T03:33:09.878071+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:21:44.928608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.894855+0000","last_change":"2024-04-13T03:31:37.587374+0000","last_active":"2024-04-13T03:33:09.894855+0000","last_peered":"2024-04-13T03:33:09.894855+0000","last_clean":"2024-04-13T03:33:09.894855+0000","last_became_active":"2024-04-13T03:31:37.586998+0000","last_became_peered":"2024-04-13T03:31:37.586998+0000","last_unstale":"2024-04-13T03:33:09.894855+0000","last_undegraded":"2024-04-13T03:33:09.894855+0000","last_fullsized":"2024-04-13T03:33:09.894855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:16:46.441001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.886660+0000","last_change":"2024-04-13T03:31:37.580579+0000","last_active":"2024-04-13T03:33:09.886660+0000","last_peered":"2024-04-13T03:33:09.886660+0000","last_clean":"2024-04-13T03:33:09.886660+0000","last_became_active":"2024-04-13T03:31:37.580243+0000","last_became_peered":"2024-04-13T03:31:37.580243+0000","last_unstale":"2024-04-13T03:33:09.886660+0000","last_undegraded":"2024-04-13T03:33:09.886660+0000","last_fullsized":"2024-04-13T03:33:09.886660+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:42.080260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.868726+0000","last_change":"2024-04-13T03:31:37.567731+0000","last_active":"2024-04-13T03:33:09.868726+0000","last_peered":"2024-04-13T03:33:09.868726+0000","last_clean":"2024-04-13T03:33:09.868726+0000","last_became_active":"2024-04-13T03:31:37.567402+0000","last_became_peered":"2024-04-13T03:31:37.567402+0000","last_unstale":"2024-04-13T03:33:09.868726+0000","last_undegraded":"2024-04-13T03:33:09.868726+0000","last_fullsized":"2024-04-13T03:33:09.868726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:54:46.976698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887191+0000","last_change":"2024-04-13T03:31:37.576379+0000","last_active":"2024-04-13T03:33:09.887191+0000","last_peered":"2024-04-13T03:33:09.887191+0000","last_clean":"2024-04-13T03:33:09.887191+0000","last_became_active":"2024-04-13T03:31:37.576062+0000","last_became_peered":"2024-04-13T03:31:37.576062+0000","last_unstale":"2024-04-13T03:33:09.887191+0000","last_undegraded":"2024-04-13T03:33:09.887191+0000","last_fullsized":"2024-04-13T03:33:09.887191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:21:17.507656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887040+0000","last_change":"2024-04-13T03:32:41.737035+0000","last_active":"2024-04-13T03:33:09.887040+0000","last_peered":"2024-04-13T03:33:09.887040+0000","last_clean":"2024-04-13T03:33:09.887040+0000","last_became_active":"2024-04-13T03:32:41.736156+0000","last_became_peered":"2024-04-13T03:32:41.736156+0000","last_unstale":"2024-04-13T03:33:09.887040+0000","last_undegraded":"2024-04-13T03:33:09.887040+0000","last_fullsized":"2024-04-13T03:33:09.887040+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:29:37.192351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.891860+0000","last_change":"2024-04-13T03:31:37.576827+0000","last_active":"2024-04-13T03:33:09.891860+0000","last_peered":"2024-04-13T03:33:09.891860+0000","last_clean":"2024-04-13T03:33:09.891860+0000","last_became_active":"2024-04-13T03:31:37.576418+0000","last_became_peered":"2024-04-13T03:31:37.576418+0000","last_unstale":"2024-04-13T03:33:09.891860+0000","last_undegraded":"2024-04-13T03:33:09.891860+0000","last_fullsized":"2024-04-13T03:33:09.891860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:45:22.462714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887378+0000","last_change":"2024-04-13T03:31:37.577623+0000","last_active":"2024-04-13T03:33:09.887378+0000","last_peered":"2024-04-13T03:33:09.887378+0000","last_clean":"2024-04-13T03:33:09.887378+0000","last_became_active":"2024-04-13T03:31:37.577175+0000","last_became_peered":"2024-04-13T03:31:37.577175+0000","last_unstale":"2024-04-13T03:33:09.887378+0000","last_undegraded":"2024-04-13T03:33:09.887378+0000","last_fullsized":"2024-04-13T03:33:09.887378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:52:29.604492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.895425+0000","last_change":"2024-04-13T03:32:41.745239+0000","last_active":"2024-04-13T03:33:09.895425+0000","last_peered":"2024-04-13T03:33:09.895425+0000","last_clean":"2024-04-13T03:33:09.895425+0000","last_became_active":"2024-04-13T03:32:41.744076+0000","last_became_peered":"2024-04-13T03:32:41.744076+0000","last_unstale":"2024-04-13T03:33:09.895425+0000","last_undegraded":"2024-04-13T03:33:09.895425+0000","last_fullsized":"2024-04-13T03:33:09.895425+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:22:50.937028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.885580+0000","last_change":"2024-04-13T03:31:37.581073+0000","last_active":"2024-04-13T03:33:09.885580+0000","last_peered":"2024-04-13T03:33:09.885580+0000","last_clean":"2024-04-13T03:33:09.885580+0000","last_became_active":"2024-04-13T03:31:37.580719+0000","last_became_peered":"2024-04-13T03:31:37.580719+0000","last_unstale":"2024-04-13T03:33:09.885580+0000","last_undegraded":"2024-04-13T03:33:09.885580+0000","last_fullsized":"2024-04-13T03:33:09.885580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:30:22.542263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.890435+0000","last_change":"2024-04-13T03:31:37.597572+0000","last_active":"2024-04-13T03:33:09.890435+0000","last_peered":"2024-04-13T03:33:09.890435+0000","last_clean":"2024-04-13T03:33:09.890435+0000","last_became_active":"2024-04-13T03:31:37.597244+0000","last_became_peered":"2024-04-13T03:31:37.597244+0000","last_unstale":"2024-04-13T03:33:09.890435+0000","last_undegraded":"2024-04-13T03:33:09.890435+0000","last_fullsized":"2024-04-13T03:33:09.890435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:39:13.365427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.885576+0000","last_change":"2024-04-13T03:31:37.596749+0000","last_active":"2024-04-13T03:33:09.885576+0000","last_peered":"2024-04-13T03:33:09.885576+0000","last_clean":"2024-04-13T03:33:09.885576+0000","last_became_active":"2024-04-13T03:31:37.592409+0000","last_became_peered":"2024-04-13T03:31:37.592409+0000","last_unstale":"2024-04-13T03:33:09.885576+0000","last_undegraded":"2024-04-13T03:33:09.885576+0000","last_fullsized":"2024-04-13T03:33:09.885576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:35:22.360224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.880477+0000","last_change":"2024-04-13T03:31:37.571354+0000","last_active":"2024-04-13T03:33:09.880477+0000","last_peered":"2024-04-13T03:33:09.880477+0000","last_clean":"2024-04-13T03:33:09.880477+0000","last_became_active":"2024-04-13T03:31:37.571012+0000","last_became_peered":"2024-04-13T03:31:37.571012+0000","last_unstale":"2024-04-13T03:33:09.880477+0000","last_undegraded":"2024-04-13T03:33:09.880477+0000","last_fullsized":"2024-04-13T03:33:09.880477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:50:26.492090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.736666+0000","last_change":"2024-04-13T03:31:41.844448+0000","last_active":"2024-04-13T03:32:41.736666+0000","last_peered":"2024-04-13T03:32:41.736666+0000","last_clean":"2024-04-13T03:32:41.736666+0000","last_became_active":"2024-04-13T03:31:41.844143+0000","last_became_peered":"2024-04-13T03:31:41.844143+0000","last_unstale":"2024-04-13T03:32:41.736666+0000","last_undegraded":"2024-04-13T03:32:41.736666+0000","last_fullsized":"2024-04-13T03:32:41.736666+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:26:40.811950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":130,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:34.991895+0000","last_change":"2024-04-13T03:31:36.581372+0000","last_active":"2024-04-13T03:33:34.991895+0000","last_peered":"2024-04-13T03:33:34.991895+0000","last_clean":"2024-04-13T03:33:34.991895+0000","last_became_active":"2024-04-13T03:31:36.580599+0000","last_became_peered":"2024-04-13T03:31:36.580599+0000","last_unstale":"2024-04-13T03:33:34.991895+0000","last_undegraded":"2024-04-13T03:33:34.991895+0000","last_fullsized":"2024-04-13T03:33:34.991895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:10:09.839049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.893115+0000","last_change":"2024-04-13T03:31:37.584511+0000","last_active":"2024-04-13T03:33:09.893115+0000","last_peered":"2024-04-13T03:33:09.893115+0000","last_clean":"2024-04-13T03:33:09.893115+0000","last_became_active":"2024-04-13T03:31:37.584094+0000","last_became_peered":"2024-04-13T03:31:37.584094+0000","last_unstale":"2024-04-13T03:33:09.893115+0000","last_undegraded":"2024-04-13T03:33:09.893115+0000","last_fullsized":"2024-04-13T03:33:09.893115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:47:36.163763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728298+0000","last_change":"2024-04-13T03:31:41.834169+0000","last_active":"2024-04-13T03:32:41.728298+0000","last_peered":"2024-04-13T03:32:41.728298+0000","last_clean":"2024-04-13T03:32:41.728298+0000","last_became_active":"2024-04-13T03:31:41.833545+0000","last_became_peered":"2024-04-13T03:31:41.833545+0000","last_unstale":"2024-04-13T03:32:41.728298+0000","last_undegraded":"2024-04-13T03:32:41.728298+0000","last_fullsized":"2024-04-13T03:32:41.728298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:37:29.657978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.908699+0000","last_change":"2024-04-13T03:31:36.596225+0000","last_active":"2024-04-13T03:32:41.908699+0000","last_peered":"2024-04-13T03:32:41.908699+0000","last_clean":"2024-04-13T03:32:41.908699+0000","last_became_active":"2024-04-13T03:31:36.595520+0000","last_became_peered":"2024-04-13T03:31:36.595520+0000","last_unstale":"2024-04-13T03:32:41.908699+0000","last_undegraded":"2024-04-13T03:32:41.908699+0000","last_fullsized":"2024-04-13T03:32:41.908699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:15:35.172065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.896101+0000","last_change":"2024-04-13T03:31:37.587264+0000","last_active":"2024-04-13T03:33:09.896101+0000","last_peered":"2024-04-13T03:33:09.896101+0000","last_clean":"2024-04-13T03:33:09.896101+0000","last_became_active":"2024-04-13T03:31:37.585574+0000","last_became_peered":"2024-04-13T03:31:37.585574+0000","last_unstale":"2024-04-13T03:33:09.896101+0000","last_undegraded":"2024-04-13T03:33:09.896101+0000","last_fullsized":"2024-04-13T03:33:09.896101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:39.258717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728520+0000","last_change":"2024-04-13T03:31:41.827237+0000","last_active":"2024-04-13T03:32:41.728520+0000","last_peered":"2024-04-13T03:32:41.728520+0000","last_clean":"2024-04-13T03:32:41.728520+0000","last_became_active":"2024-04-13T03:31:41.826799+0000","last_became_peered":"2024-04-13T03:31:41.826799+0000","last_unstale":"2024-04-13T03:32:41.728520+0000","last_undegraded":"2024-04-13T03:32:41.728520+0000","last_fullsized":"2024-04-13T03:32:41.728520+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:26:14.337654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728623+0000","last_change":"2024-04-13T03:31:36.564752+0000","last_active":"2024-04-13T03:32:41.728623+0000","last_peered":"2024-04-13T03:32:41.728623+0000","last_clean":"2024-04-13T03:32:41.728623+0000","last_became_active":"2024-04-13T03:31:36.564413+0000","last_became_peered":"2024-04-13T03:31:36.564413+0000","last_unstale":"2024-04-13T03:32:41.728623+0000","last_undegraded":"2024-04-13T03:32:41.728623+0000","last_fullsized":"2024-04-13T03:32:41.728623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:34:52.369875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.890659+0000","last_change":"2024-04-13T03:31:37.579316+0000","last_active":"2024-04-13T03:33:09.890659+0000","last_peered":"2024-04-13T03:33:09.890659+0000","last_clean":"2024-04-13T03:33:09.890659+0000","last_became_active":"2024-04-13T03:31:37.578971+0000","last_became_peered":"2024-04-13T03:31:37.578971+0000","last_unstale":"2024-04-13T03:33:09.890659+0000","last_undegraded":"2024-04-13T03:33:09.890659+0000","last_fullsized":"2024-04-13T03:33:09.890659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:54:57.013722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728358+0000","last_change":"2024-04-13T03:31:41.841120+0000","last_active":"2024-04-13T03:32:41.728358+0000","last_peered":"2024-04-13T03:32:41.728358+0000","last_clean":"2024-04-13T03:32:41.728358+0000","last_became_active":"2024-04-13T03:31:41.840833+0000","last_became_peered":"2024-04-13T03:31:41.840833+0000","last_unstale":"2024-04-13T03:32:41.728358+0000","last_undegraded":"2024-04-13T03:32:41.728358+0000","last_fullsized":"2024-04-13T03:32:41.728358+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:13:09.979055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728913+0000","last_change":"2024-04-13T03:31:36.561428+0000","last_active":"2024-04-13T03:32:41.728913+0000","last_peered":"2024-04-13T03:32:41.728913+0000","last_clean":"2024-04-13T03:32:41.728913+0000","last_became_active":"2024-04-13T03:31:36.561100+0000","last_became_peered":"2024-04-13T03:31:36.561100+0000","last_unstale":"2024-04-13T03:32:41.728913+0000","last_undegraded":"2024-04-13T03:32:41.728913+0000","last_fullsized":"2024-04-13T03:32:41.728913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:04:32.125292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.885918+0000","last_change":"2024-04-13T03:32:41.736642+0000","last_active":"2024-04-13T03:33:09.885918+0000","last_peered":"2024-04-13T03:33:09.885918+0000","last_clean":"2024-04-13T03:33:09.885918+0000","last_became_active":"2024-04-13T03:32:41.736237+0000","last_became_peered":"2024-04-13T03:32:41.736237+0000","last_unstale":"2024-04-13T03:33:09.885918+0000","last_undegraded":"2024-04-13T03:33:09.885918+0000","last_fullsized":"2024-04-13T03:33:09.885918+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:10:14.027179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.737603+0000","last_change":"2024-04-13T03:31:41.824295+0000","last_active":"2024-04-13T03:32:41.737603+0000","last_peered":"2024-04-13T03:32:41.737603+0000","last_clean":"2024-04-13T03:32:41.737603+0000","last_became_active":"2024-04-13T03:31:41.823810+0000","last_became_peered":"2024-04-13T03:31:41.823810+0000","last_unstale":"2024-04-13T03:32:41.737603+0000","last_undegraded":"2024-04-13T03:32:41.737603+0000","last_fullsized":"2024-04-13T03:32:41.737603+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:40:08.888519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":124,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:32.989376+0000","last_change":"2024-04-13T03:31:36.558866+0000","last_active":"2024-04-13T03:33:32.989376+0000","last_peered":"2024-04-13T03:33:32.989376+0000","last_clean":"2024-04-13T03:33:32.989376+0000","last_became_active":"2024-04-13T03:31:36.558541+0000","last_became_peered":"2024-04-13T03:31:36.558541+0000","last_unstale":"2024-04-13T03:33:32.989376+0000","last_undegraded":"2024-04-13T03:33:32.989376+0000","last_fullsized":"2024-04-13T03:33:32.989376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:59:29.431084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.882838+0000","last_change":"2024-04-13T03:31:37.581298+0000","last_active":"2024-04-13T03:33:09.882838+0000","last_peered":"2024-04-13T03:33:09.882838+0000","last_clean":"2024-04-13T03:33:09.882838+0000","last_became_active":"2024-04-13T03:31:37.580963+0000","last_became_peered":"2024-04-13T03:31:37.580963+0000","last_unstale":"2024-04-13T03:33:09.882838+0000","last_undegraded":"2024-04-13T03:33:09.882838+0000","last_fullsized":"2024-04-13T03:33:09.882838+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:47:27.921928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.737546+0000","last_change":"2024-04-13T03:31:41.831221+0000","last_active":"2024-04-13T03:32:41.737546+0000","last_peered":"2024-04-13T03:32:41.737546+0000","last_clean":"2024-04-13T03:32:41.737546+0000","last_became_active":"2024-04-13T03:31:41.830246+0000","last_became_peered":"2024-04-13T03:31:41.830246+0000","last_unstale":"2024-04-13T03:32:41.737546+0000","last_undegraded":"2024-04-13T03:32:41.737546+0000","last_fullsized":"2024-04-13T03:32:41.737546+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:08:16.624658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.733886+0000","last_change":"2024-04-13T03:31:36.585534+0000","last_active":"2024-04-13T03:32:41.733886+0000","last_peered":"2024-04-13T03:32:41.733886+0000","last_clean":"2024-04-13T03:32:41.733886+0000","last_became_active":"2024-04-13T03:31:36.585324+0000","last_became_peered":"2024-04-13T03:31:36.585324+0000","last_unstale":"2024-04-13T03:32:41.733886+0000","last_undegraded":"2024-04-13T03:32:41.733886+0000","last_fullsized":"2024-04-13T03:32:41.733886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:12:35.971994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.885000+0000","last_change":"2024-04-13T03:31:37.587440+0000","last_active":"2024-04-13T03:33:09.885000+0000","last_peered":"2024-04-13T03:33:09.885000+0000","last_clean":"2024-04-13T03:33:09.885000+0000","last_became_active":"2024-04-13T03:31:37.581413+0000","last_became_peered":"2024-04-13T03:31:37.581413+0000","last_unstale":"2024-04-13T03:33:09.885000+0000","last_undegraded":"2024-04-13T03:33:09.885000+0000","last_fullsized":"2024-04-13T03:33:09.885000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:14:26.066107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731161+0000","last_change":"2024-04-13T03:31:41.848451+0000","last_active":"2024-04-13T03:32:41.731161+0000","last_peered":"2024-04-13T03:32:41.731161+0000","last_clean":"2024-04-13T03:32:41.731161+0000","last_became_active":"2024-04-13T03:31:41.848019+0000","last_became_peered":"2024-04-13T03:31:41.848019+0000","last_unstale":"2024-04-13T03:32:41.731161+0000","last_undegraded":"2024-04-13T03:32:41.731161+0000","last_fullsized":"2024-04-13T03:32:41.731161+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:26:34.558142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741735+0000","last_change":"2024-04-13T03:31:36.588393+0000","last_active":"2024-04-13T03:32:41.741735+0000","last_peered":"2024-04-13T03:32:41.741735+0000","last_clean":"2024-04-13T03:32:41.741735+0000","last_became_active":"2024-04-13T03:31:36.587184+0000","last_became_peered":"2024-04-13T03:31:36.587184+0000","last_unstale":"2024-04-13T03:32:41.741735+0000","last_undegraded":"2024-04-13T03:32:41.741735+0000","last_fullsized":"2024-04-13T03:32:41.741735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:44:24.985634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.892252+0000","last_change":"2024-04-13T03:31:37.600417+0000","last_active":"2024-04-13T03:33:09.892252+0000","last_peered":"2024-04-13T03:33:09.892252+0000","last_clean":"2024-04-13T03:33:09.892252+0000","last_became_active":"2024-04-13T03:31:37.599889+0000","last_became_peered":"2024-04-13T03:31:37.599889+0000","last_unstale":"2024-04-13T03:33:09.892252+0000","last_undegraded":"2024-04-13T03:33:09.892252+0000","last_fullsized":"2024-04-13T03:33:09.892252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:19:13.039127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.735636+0000","last_change":"2024-04-13T03:31:41.848432+0000","last_active":"2024-04-13T03:32:41.735636+0000","last_peered":"2024-04-13T03:32:41.735636+0000","last_clean":"2024-04-13T03:32:41.735636+0000","last_became_active":"2024-04-13T03:31:41.843515+0000","last_became_peered":"2024-04-13T03:31:41.843515+0000","last_unstale":"2024-04-13T03:32:41.735636+0000","last_undegraded":"2024-04-13T03:32:41.735636+0000","last_fullsized":"2024-04-13T03:32:41.735636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:31:15.512541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.733101+0000","last_change":"2024-04-13T03:31:36.560090+0000","last_active":"2024-04-13T03:32:41.733101+0000","last_peered":"2024-04-13T03:32:41.733101+0000","last_clean":"2024-04-13T03:32:41.733101+0000","last_became_active":"2024-04-13T03:31:36.559770+0000","last_became_peered":"2024-04-13T03:31:36.559770+0000","last_unstale":"2024-04-13T03:32:41.733101+0000","last_undegraded":"2024-04-13T03:32:41.733101+0000","last_fullsized":"2024-04-13T03:32:41.733101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:45:51.715456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.888658+0000","last_change":"2024-04-13T03:31:37.593845+0000","last_active":"2024-04-13T03:33:09.888658+0000","last_peered":"2024-04-13T03:33:09.888658+0000","last_clean":"2024-04-13T03:33:09.888658+0000","last_became_active":"2024-04-13T03:31:37.593484+0000","last_became_peered":"2024-04-13T03:31:37.593484+0000","last_unstale":"2024-04-13T03:33:09.888658+0000","last_undegraded":"2024-04-13T03:33:09.888658+0000","last_fullsized":"2024-04-13T03:33:09.888658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:38:05.817746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.908619+0000","last_change":"2024-04-13T03:31:41.830949+0000","last_active":"2024-04-13T03:32:41.908619+0000","last_peered":"2024-04-13T03:32:41.908619+0000","last_clean":"2024-04-13T03:32:41.908619+0000","last_became_active":"2024-04-13T03:31:41.830540+0000","last_became_peered":"2024-04-13T03:31:41.830540+0000","last_unstale":"2024-04-13T03:32:41.908619+0000","last_undegraded":"2024-04-13T03:32:41.908619+0000","last_fullsized":"2024-04-13T03:32:41.908619+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:09:00.006158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'5","reported_seq":206,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:33.935591+0000","last_change":"2024-04-13T03:31:36.574773+0000","last_active":"2024-04-13T03:33:33.935591+0000","last_peered":"2024-04-13T03:33:33.935591+0000","last_clean":"2024-04-13T03:33:33.935591+0000","last_became_active":"2024-04-13T03:31:36.574511+0000","last_became_peered":"2024-04-13T03:31:36.574511+0000","last_unstale":"2024-04-13T03:33:33.935591+0000","last_undegraded":"2024-04-13T03:33:33.935591+0000","last_fullsized":"2024-04-13T03:33:33.935591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:26:41.010269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":174,"num_read_kb":174,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.883268+0000","last_change":"2024-04-13T03:31:37.583383+0000","last_active":"2024-04-13T03:33:09.883268+0000","last_peered":"2024-04-13T03:33:09.883268+0000","last_clean":"2024-04-13T03:33:09.883268+0000","last_became_active":"2024-04-13T03:31:37.583026+0000","last_became_peered":"2024-04-13T03:31:37.583026+0000","last_unstale":"2024-04-13T03:33:09.883268+0000","last_undegraded":"2024-04-13T03:33:09.883268+0000","last_fullsized":"2024-04-13T03:33:09.883268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T15:16:34.790848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728631+0000","last_change":"2024-04-13T03:31:41.845867+0000","last_active":"2024-04-13T03:32:41.728631+0000","last_peered":"2024-04-13T03:32:41.728631+0000","last_clean":"2024-04-13T03:32:41.728631+0000","last_became_active":"2024-04-13T03:31:41.845408+0000","last_became_peered":"2024-04-13T03:31:41.845408+0000","last_unstale":"2024-04-13T03:32:41.728631+0000","last_undegraded":"2024-04-13T03:32:41.728631+0000","last_fullsized":"2024-04-13T03:32:41.728631+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:10:42.830683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.730799+0000","last_change":"2024-04-13T03:31:36.562534+0000","last_active":"2024-04-13T03:32:41.730799+0000","last_peered":"2024-04-13T03:32:41.730799+0000","last_clean":"2024-04-13T03:32:41.730799+0000","last_became_active":"2024-04-13T03:31:36.562189+0000","last_became_peered":"2024-04-13T03:31:36.562189+0000","last_unstale":"2024-04-13T03:32:41.730799+0000","last_undegraded":"2024-04-13T03:32:41.730799+0000","last_fullsized":"2024-04-13T03:32:41.730799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:41:27.504982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.890232+0000","last_change":"2024-04-13T03:31:37.588637+0000","last_active":"2024-04-13T03:33:09.890232+0000","last_peered":"2024-04-13T03:33:09.890232+0000","last_clean":"2024-04-13T03:33:09.890232+0000","last_became_active":"2024-04-13T03:31:37.583263+0000","last_became_peered":"2024-04-13T03:31:37.583263+0000","last_unstale":"2024-04-13T03:33:09.890232+0000","last_undegraded":"2024-04-13T03:33:09.890232+0000","last_fullsized":"2024-04-13T03:33:09.890232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:07:40.503310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728659+0000","last_change":"2024-04-13T03:31:41.827302+0000","last_active":"2024-04-13T03:32:41.728659+0000","last_peered":"2024-04-13T03:32:41.728659+0000","last_clean":"2024-04-13T03:32:41.728659+0000","last_became_active":"2024-04-13T03:31:41.826822+0000","last_became_peered":"2024-04-13T03:31:41.826822+0000","last_unstale":"2024-04-13T03:32:41.728659+0000","last_undegraded":"2024-04-13T03:32:41.728659+0000","last_fullsized":"2024-04-13T03:32:41.728659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:00:16.589319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732293+0000","last_change":"2024-04-13T03:31:36.582710+0000","last_active":"2024-04-13T03:32:41.732293+0000","last_peered":"2024-04-13T03:32:41.732293+0000","last_clean":"2024-04-13T03:32:41.732293+0000","last_became_active":"2024-04-13T03:31:36.582372+0000","last_became_peered":"2024-04-13T03:31:36.582372+0000","last_unstale":"2024-04-13T03:32:41.732293+0000","last_undegraded":"2024-04-13T03:32:41.732293+0000","last_fullsized":"2024-04-13T03:32:41.732293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:38:58.990256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.886658+0000","last_change":"2024-04-13T03:31:37.578345+0000","last_active":"2024-04-13T03:33:09.886658+0000","last_peered":"2024-04-13T03:33:09.886658+0000","last_clean":"2024-04-13T03:33:09.886658+0000","last_became_active":"2024-04-13T03:31:37.578026+0000","last_became_peered":"2024-04-13T03:31:37.578026+0000","last_unstale":"2024-04-13T03:33:09.886658+0000","last_undegraded":"2024-04-13T03:33:09.886658+0000","last_fullsized":"2024-04-13T03:33:09.886658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:59:02.080243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.729113+0000","last_change":"2024-04-13T03:31:41.829108+0000","last_active":"2024-04-13T03:32:41.729113+0000","last_peered":"2024-04-13T03:32:41.729113+0000","last_clean":"2024-04-13T03:32:41.729113+0000","last_became_active":"2024-04-13T03:31:41.828603+0000","last_became_peered":"2024-04-13T03:31:41.828603+0000","last_unstale":"2024-04-13T03:32:41.729113+0000","last_undegraded":"2024-04-13T03:32:41.729113+0000","last_fullsized":"2024-04-13T03:32:41.729113+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:06:32.753928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'38","reported_seq":265,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:33.825923+0000","last_change":"2024-04-13T03:32:41.746579+0000","last_active":"2024-04-13T03:33:33.825923+0000","last_peered":"2024-04-13T03:33:33.825923+0000","last_clean":"2024-04-13T03:33:33.825923+0000","last_became_active":"2024-04-13T03:32:41.737639+0000","last_became_peered":"2024-04-13T03:32:41.737639+0000","last_unstale":"2024-04-13T03:33:33.825923+0000","last_undegraded":"2024-04-13T03:33:33.825923+0000","last_fullsized":"2024-04-13T03:33:33.825923+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:15:09.694242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":180,"num_read_kb":153,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":64206,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.860182+0000","last_change":"2024-04-13T03:31:37.580515+0000","last_active":"2024-04-13T03:33:09.860182+0000","last_peered":"2024-04-13T03:33:09.860182+0000","last_clean":"2024-04-13T03:33:09.860182+0000","last_became_active":"2024-04-13T03:31:37.580157+0000","last_became_peered":"2024-04-13T03:31:37.580157+0000","last_unstale":"2024-04-13T03:33:09.860182+0000","last_undegraded":"2024-04-13T03:33:09.860182+0000","last_fullsized":"2024-04-13T03:33:09.860182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:17:17.843273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732587+0000","last_change":"2024-04-13T03:31:41.833555+0000","last_active":"2024-04-13T03:32:41.732587+0000","last_peered":"2024-04-13T03:32:41.732587+0000","last_clean":"2024-04-13T03:32:41.732587+0000","last_became_active":"2024-04-13T03:31:41.833134+0000","last_became_peered":"2024-04-13T03:31:41.833134+0000","last_unstale":"2024-04-13T03:32:41.732587+0000","last_undegraded":"2024-04-13T03:32:41.732587+0000","last_fullsized":"2024-04-13T03:32:41.732587+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:22:58.311184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.737030+0000","last_change":"2024-04-13T03:31:36.591127+0000","last_active":"2024-04-13T03:32:41.737030+0000","last_peered":"2024-04-13T03:32:41.737030+0000","last_clean":"2024-04-13T03:32:41.737030+0000","last_became_active":"2024-04-13T03:31:36.590819+0000","last_became_peered":"2024-04-13T03:31:36.590819+0000","last_unstale":"2024-04-13T03:32:41.737030+0000","last_undegraded":"2024-04-13T03:32:41.737030+0000","last_fullsized":"2024-04-13T03:32:41.737030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:54:51.637340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.888324+0000","last_change":"2024-04-13T03:32:41.744284+0000","last_active":"2024-04-13T03:33:09.888324+0000","last_peered":"2024-04-13T03:33:09.888324+0000","last_clean":"2024-04-13T03:33:09.888324+0000","last_became_active":"2024-04-13T03:32:41.743848+0000","last_became_peered":"2024-04-13T03:32:41.743848+0000","last_unstale":"2024-04-13T03:33:09.888324+0000","last_undegraded":"2024-04-13T03:33:09.888324+0000","last_fullsized":"2024-04-13T03:33:09.888324+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:11:23.967329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731075+0000","last_change":"2024-04-13T03:31:41.833559+0000","last_active":"2024-04-13T03:32:41.731075+0000","last_peered":"2024-04-13T03:32:41.731075+0000","last_clean":"2024-04-13T03:32:41.731075+0000","last_became_active":"2024-04-13T03:31:41.833067+0000","last_became_peered":"2024-04-13T03:31:41.833067+0000","last_unstale":"2024-04-13T03:32:41.731075+0000","last_undegraded":"2024-04-13T03:32:41.731075+0000","last_fullsized":"2024-04-13T03:32:41.731075+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:56:09.027293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732615+0000","last_change":"2024-04-13T03:31:36.597136+0000","last_active":"2024-04-13T03:32:41.732615+0000","last_peered":"2024-04-13T03:32:41.732615+0000","last_clean":"2024-04-13T03:32:41.732615+0000","last_became_active":"2024-04-13T03:31:36.596789+0000","last_became_peered":"2024-04-13T03:31:36.596789+0000","last_unstale":"2024-04-13T03:32:41.732615+0000","last_undegraded":"2024-04-13T03:32:41.732615+0000","last_fullsized":"2024-04-13T03:32:41.732615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:39:28.371669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.893295+0000","last_change":"2024-04-13T03:31:37.584424+0000","last_active":"2024-04-13T03:33:09.893295+0000","last_peered":"2024-04-13T03:33:09.893295+0000","last_clean":"2024-04-13T03:33:09.893295+0000","last_became_active":"2024-04-13T03:31:37.584125+0000","last_became_peered":"2024-04-13T03:31:37.584125+0000","last_unstale":"2024-04-13T03:33:09.893295+0000","last_undegraded":"2024-04-13T03:33:09.893295+0000","last_fullsized":"2024-04-13T03:33:09.893295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:07:53.625486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731385+0000","last_change":"2024-04-13T03:31:41.852755+0000","last_active":"2024-04-13T03:32:41.731385+0000","last_peered":"2024-04-13T03:32:41.731385+0000","last_clean":"2024-04-13T03:32:41.731385+0000","last_became_active":"2024-04-13T03:31:41.851975+0000","last_became_peered":"2024-04-13T03:31:41.851975+0000","last_unstale":"2024-04-13T03:32:41.731385+0000","last_undegraded":"2024-04-13T03:32:41.731385+0000","last_fullsized":"2024-04-13T03:32:41.731385+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:19:40.982525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"92'7","reported_seq":126,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:35.012187+0000","last_change":"2024-04-13T03:31:36.559351+0000","last_active":"2024-04-13T03:33:35.012187+0000","last_peered":"2024-04-13T03:33:35.012187+0000","last_clean":"2024-04-13T03:33:35.012187+0000","last_became_active":"2024-04-13T03:31:36.559011+0000","last_became_peered":"2024-04-13T03:31:36.559011+0000","last_unstale":"2024-04-13T03:33:35.012187+0000","last_undegraded":"2024-04-13T03:33:35.012187+0000","last_fullsized":"2024-04-13T03:33:35.012187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:08:08.983088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.878079+0000","last_change":"2024-04-13T03:31:37.580884+0000","last_active":"2024-04-13T03:33:09.878079+0000","last_peered":"2024-04-13T03:33:09.878079+0000","last_clean":"2024-04-13T03:33:09.878079+0000","last_became_active":"2024-04-13T03:31:37.580568+0000","last_became_peered":"2024-04-13T03:31:37.580568+0000","last_unstale":"2024-04-13T03:33:09.878079+0000","last_undegraded":"2024-04-13T03:33:09.878079+0000","last_fullsized":"2024-04-13T03:33:09.878079+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:42:41.484614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741856+0000","last_change":"2024-04-13T03:31:41.840912+0000","last_active":"2024-04-13T03:32:41.741856+0000","last_peered":"2024-04-13T03:32:41.741856+0000","last_clean":"2024-04-13T03:32:41.741856+0000","last_became_active":"2024-04-13T03:31:41.840129+0000","last_became_peered":"2024-04-13T03:31:41.840129+0000","last_unstale":"2024-04-13T03:32:41.741856+0000","last_undegraded":"2024-04-13T03:32:41.741856+0000","last_fullsized":"2024-04-13T03:32:41.741856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:45:33.716345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728411+0000","last_change":"2024-04-13T03:31:36.580230+0000","last_active":"2024-04-13T03:32:41.728411+0000","last_peered":"2024-04-13T03:32:41.728411+0000","last_clean":"2024-04-13T03:32:41.728411+0000","last_became_active":"2024-04-13T03:31:36.579874+0000","last_became_peered":"2024-04-13T03:31:36.579874+0000","last_unstale":"2024-04-13T03:32:41.728411+0000","last_undegraded":"2024-04-13T03:32:41.728411+0000","last_fullsized":"2024-04-13T03:32:41.728411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:30:17.154748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.882696+0000","last_change":"2024-04-13T03:31:37.581937+0000","last_active":"2024-04-13T03:33:09.882696+0000","last_peered":"2024-04-13T03:33:09.882696+0000","last_clean":"2024-04-13T03:33:09.882696+0000","last_became_active":"2024-04-13T03:31:37.581547+0000","last_became_peered":"2024-04-13T03:31:37.581547+0000","last_unstale":"2024-04-13T03:33:09.882696+0000","last_undegraded":"2024-04-13T03:33:09.882696+0000","last_fullsized":"2024-04-13T03:33:09.882696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:19:28.600996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732541+0000","last_change":"2024-04-13T03:31:41.842887+0000","last_active":"2024-04-13T03:32:41.732541+0000","last_peered":"2024-04-13T03:32:41.732541+0000","last_clean":"2024-04-13T03:32:41.732541+0000","last_became_active":"2024-04-13T03:31:41.842537+0000","last_became_peered":"2024-04-13T03:31:41.842537+0000","last_unstale":"2024-04-13T03:32:41.732541+0000","last_undegraded":"2024-04-13T03:32:41.732541+0000","last_fullsized":"2024-04-13T03:32:41.732541+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:44:38.930759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.729018+0000","last_change":"2024-04-13T03:31:36.561347+0000","last_active":"2024-04-13T03:32:41.729018+0000","last_peered":"2024-04-13T03:32:41.729018+0000","last_clean":"2024-04-13T03:32:41.729018+0000","last_became_active":"2024-04-13T03:31:36.560969+0000","last_became_peered":"2024-04-13T03:31:36.560969+0000","last_unstale":"2024-04-13T03:32:41.729018+0000","last_undegraded":"2024-04-13T03:32:41.729018+0000","last_fullsized":"2024-04-13T03:32:41.729018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:59:38.825930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.869287+0000","last_change":"2024-04-13T03:31:37.587599+0000","last_active":"2024-04-13T03:33:09.869287+0000","last_peered":"2024-04-13T03:33:09.869287+0000","last_clean":"2024-04-13T03:33:09.869287+0000","last_became_active":"2024-04-13T03:31:37.587257+0000","last_became_peered":"2024-04-13T03:31:37.587257+0000","last_unstale":"2024-04-13T03:33:09.869287+0000","last_undegraded":"2024-04-13T03:33:09.869287+0000","last_fullsized":"2024-04-13T03:33:09.869287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:39:57.956174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728661+0000","last_change":"2024-04-13T03:31:41.831134+0000","last_active":"2024-04-13T03:32:41.728661+0000","last_peered":"2024-04-13T03:32:41.728661+0000","last_clean":"2024-04-13T03:32:41.728661+0000","last_became_active":"2024-04-13T03:31:41.830671+0000","last_became_peered":"2024-04-13T03:31:41.830671+0000","last_unstale":"2024-04-13T03:32:41.728661+0000","last_undegraded":"2024-04-13T03:32:41.728661+0000","last_fullsized":"2024-04-13T03:32:41.728661+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:50:59.527951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731300+0000","last_change":"2024-04-13T03:31:36.586008+0000","last_active":"2024-04-13T03:32:41.731300+0000","last_peered":"2024-04-13T03:32:41.731300+0000","last_clean":"2024-04-13T03:32:41.731300+0000","last_became_active":"2024-04-13T03:31:36.583937+0000","last_became_peered":"2024-04-13T03:31:36.583937+0000","last_unstale":"2024-04-13T03:32:41.731300+0000","last_undegraded":"2024-04-13T03:32:41.731300+0000","last_fullsized":"2024-04-13T03:32:41.731300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:17:06.194397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.888950+0000","last_change":"2024-04-13T03:31:37.586185+0000","last_active":"2024-04-13T03:33:09.888950+0000","last_peered":"2024-04-13T03:33:09.888950+0000","last_clean":"2024-04-13T03:33:09.888950+0000","last_became_active":"2024-04-13T03:31:37.585509+0000","last_became_peered":"2024-04-13T03:31:37.585509+0000","last_unstale":"2024-04-13T03:33:09.888950+0000","last_undegraded":"2024-04-13T03:33:09.888950+0000","last_fullsized":"2024-04-13T03:33:09.888950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:35:41.064317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.740694+0000","last_change":"2024-04-13T03:31:41.847376+0000","last_active":"2024-04-13T03:32:41.740694+0000","last_peered":"2024-04-13T03:32:41.740694+0000","last_clean":"2024-04-13T03:32:41.740694+0000","last_became_active":"2024-04-13T03:31:41.846724+0000","last_became_peered":"2024-04-13T03:31:41.846724+0000","last_unstale":"2024-04-13T03:32:41.740694+0000","last_undegraded":"2024-04-13T03:32:41.740694+0000","last_fullsized":"2024-04-13T03:32:41.740694+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:26:14.046875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":122,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:34.992627+0000","last_change":"2024-04-13T03:31:36.582776+0000","last_active":"2024-04-13T03:33:34.992627+0000","last_peered":"2024-04-13T03:33:34.992627+0000","last_clean":"2024-04-13T03:33:34.992627+0000","last_became_active":"2024-04-13T03:31:36.582405+0000","last_became_peered":"2024-04-13T03:31:36.582405+0000","last_unstale":"2024-04-13T03:33:34.992627+0000","last_undegraded":"2024-04-13T03:33:34.992627+0000","last_fullsized":"2024-04-13T03:33:34.992627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:18:56.624442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.866497+0000","last_change":"2024-04-13T03:31:37.580552+0000","last_active":"2024-04-13T03:33:09.866497+0000","last_peered":"2024-04-13T03:33:09.866497+0000","last_clean":"2024-04-13T03:33:09.866497+0000","last_became_active":"2024-04-13T03:31:37.576390+0000","last_became_peered":"2024-04-13T03:31:37.576390+0000","last_unstale":"2024-04-13T03:33:09.866497+0000","last_undegraded":"2024-04-13T03:33:09.866497+0000","last_fullsized":"2024-04-13T03:33:09.866497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:50:48.328048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741384+0000","last_change":"2024-04-13T03:31:41.849818+0000","last_active":"2024-04-13T03:32:41.741384+0000","last_peered":"2024-04-13T03:32:41.741384+0000","last_clean":"2024-04-13T03:32:41.741384+0000","last_became_active":"2024-04-13T03:31:41.848846+0000","last_became_peered":"2024-04-13T03:31:41.848846+0000","last_unstale":"2024-04-13T03:32:41.741384+0000","last_undegraded":"2024-04-13T03:32:41.741384+0000","last_fullsized":"2024-04-13T03:32:41.741384+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:55:16.764935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.868852+0000","last_change":"2024-04-13T03:31:37.581723+0000","last_active":"2024-04-13T03:33:09.868852+0000","last_peered":"2024-04-13T03:33:09.868852+0000","last_clean":"2024-04-13T03:33:09.868852+0000","last_became_active":"2024-04-13T03:31:37.578857+0000","last_became_peered":"2024-04-13T03:31:37.578857+0000","last_unstale":"2024-04-13T03:33:09.868852+0000","last_undegraded":"2024-04-13T03:33:09.868852+0000","last_fullsized":"2024-04-13T03:33:09.868852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:47:34.447256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'6","reported_seq":120,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:32.003033+0000","last_change":"2024-04-13T03:31:36.592275+0000","last_active":"2024-04-13T03:33:32.003033+0000","last_peered":"2024-04-13T03:33:32.003033+0000","last_clean":"2024-04-13T03:33:32.003033+0000","last_became_active":"2024-04-13T03:31:36.591905+0000","last_became_peered":"2024-04-13T03:31:36.591905+0000","last_unstale":"2024-04-13T03:33:32.003033+0000","last_undegraded":"2024-04-13T03:33:32.003033+0000","last_fullsized":"2024-04-13T03:33:32.003033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:42:42.591189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.730575+0000","last_change":"2024-04-13T03:31:41.835412+0000","last_active":"2024-04-13T03:32:41.730575+0000","last_peered":"2024-04-13T03:32:41.730575+0000","last_clean":"2024-04-13T03:32:41.730575+0000","last_became_active":"2024-04-13T03:31:41.834978+0000","last_became_peered":"2024-04-13T03:31:41.834978+0000","last_unstale":"2024-04-13T03:32:41.730575+0000","last_undegraded":"2024-04-13T03:32:41.730575+0000","last_fullsized":"2024-04-13T03:32:41.730575+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:45:52.037452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887114+0000","last_change":"2024-04-13T03:32:41.736526+0000","last_active":"2024-04-13T03:33:09.887114+0000","last_peered":"2024-04-13T03:33:09.887114+0000","last_clean":"2024-04-13T03:33:09.887114+0000","last_became_active":"2024-04-13T03:32:41.736021+0000","last_became_peered":"2024-04-13T03:32:41.736021+0000","last_unstale":"2024-04-13T03:33:09.887114+0000","last_undegraded":"2024-04-13T03:33:09.887114+0000","last_fullsized":"2024-04-13T03:33:09.887114+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:54:13.855368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728528+0000","last_change":"2024-04-13T03:31:36.580058+0000","last_active":"2024-04-13T03:32:41.728528+0000","last_peered":"2024-04-13T03:32:41.728528+0000","last_clean":"2024-04-13T03:32:41.728528+0000","last_became_active":"2024-04-13T03:31:36.579777+0000","last_became_peered":"2024-04-13T03:31:36.579777+0000","last_unstale":"2024-04-13T03:32:41.728528+0000","last_undegraded":"2024-04-13T03:32:41.728528+0000","last_fullsized":"2024-04-13T03:32:41.728528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:27:50.693295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":362,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.730653+0000","last_change":"2024-04-13T03:30:17.781014+0000","last_active":"2024-04-13T03:32:41.730653+0000","last_peered":"2024-04-13T03:32:41.730653+0000","last_clean":"2024-04-13T03:32:41.730653+0000","last_became_active":"2024-04-13T03:30:17.780417+0000","last_became_peered":"2024-04-13T03:30:17.780417+0000","last_unstale":"2024-04-13T03:32:41.730653+0000","last_undegraded":"2024-04-13T03:32:41.730653+0000","last_fullsized":"2024-04-13T03:32:41.730653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:26:22.174408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:26:22.174408+0000","last_clean_scrub_stamp":"2024-04-13T03:26:22.174408+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:52:50.418399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732734+0000","last_change":"2024-04-13T03:31:41.837973+0000","last_active":"2024-04-13T03:32:41.732734+0000","last_peered":"2024-04-13T03:32:41.732734+0000","last_clean":"2024-04-13T03:32:41.732734+0000","last_became_active":"2024-04-13T03:31:41.837562+0000","last_became_peered":"2024-04-13T03:31:41.837562+0000","last_unstale":"2024-04-13T03:32:41.732734+0000","last_undegraded":"2024-04-13T03:32:41.732734+0000","last_fullsized":"2024-04-13T03:32:41.732734+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:05:09.900193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.884631+0000","last_change":"2024-04-13T03:31:37.584279+0000","last_active":"2024-04-13T03:33:09.884631+0000","last_peered":"2024-04-13T03:33:09.884631+0000","last_clean":"2024-04-13T03:33:09.884631+0000","last_became_active":"2024-04-13T03:31:37.583891+0000","last_became_peered":"2024-04-13T03:31:37.583891+0000","last_unstale":"2024-04-13T03:33:09.884631+0000","last_undegraded":"2024-04-13T03:33:09.884631+0000","last_fullsized":"2024-04-13T03:33:09.884631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:56:21.446525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741621+0000","last_change":"2024-04-13T03:32:41.741100+0000","last_active":"2024-04-13T03:32:41.741621+0000","last_peered":"2024-04-13T03:32:41.741621+0000","last_clean":"2024-04-13T03:32:41.741621+0000","last_became_active":"2024-04-13T03:32:41.735297+0000","last_became_peered":"2024-04-13T03:32:41.735297+0000","last_unstale":"2024-04-13T03:32:41.741621+0000","last_undegraded":"2024-04-13T03:32:41.741621+0000","last_fullsized":"2024-04-13T03:32:41.741621+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:00:42.088105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731004+0000","last_change":"2024-04-13T03:31:41.833350+0000","last_active":"2024-04-13T03:32:41.731004+0000","last_peered":"2024-04-13T03:32:41.731004+0000","last_clean":"2024-04-13T03:32:41.731004+0000","last_became_active":"2024-04-13T03:31:41.832865+0000","last_became_peered":"2024-04-13T03:31:41.832865+0000","last_unstale":"2024-04-13T03:32:41.731004+0000","last_undegraded":"2024-04-13T03:32:41.731004+0000","last_fullsized":"2024-04-13T03:32:41.731004+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:23:31.872562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.751140+0000","last_change":"2024-04-13T03:32:41.750758+0000","last_active":"2024-04-13T03:32:41.751140+0000","last_peered":"2024-04-13T03:32:41.751140+0000","last_clean":"2024-04-13T03:32:41.751140+0000","last_became_active":"2024-04-13T03:32:41.740448+0000","last_became_peered":"2024-04-13T03:32:41.740448+0000","last_unstale":"2024-04-13T03:32:41.751140+0000","last_undegraded":"2024-04-13T03:32:41.751140+0000","last_fullsized":"2024-04-13T03:32:41.751140+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:07:30.563468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.884214+0000","last_change":"2024-04-13T03:31:37.584429+0000","last_active":"2024-04-13T03:33:09.884214+0000","last_peered":"2024-04-13T03:33:09.884214+0000","last_clean":"2024-04-13T03:33:09.884214+0000","last_became_active":"2024-04-13T03:31:37.583811+0000","last_became_peered":"2024-04-13T03:31:37.583811+0000","last_unstale":"2024-04-13T03:33:09.884214+0000","last_undegraded":"2024-04-13T03:33:09.884214+0000","last_fullsized":"2024-04-13T03:33:09.884214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:52:20.227001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.907911+0000","last_change":"2024-04-13T03:31:41.846065+0000","last_active":"2024-04-13T03:32:41.907911+0000","last_peered":"2024-04-13T03:32:41.907911+0000","last_clean":"2024-04-13T03:32:41.907911+0000","last_became_active":"2024-04-13T03:31:41.844422+0000","last_became_peered":"2024-04-13T03:31:41.844422+0000","last_unstale":"2024-04-13T03:32:41.907911+0000","last_undegraded":"2024-04-13T03:32:41.907911+0000","last_fullsized":"2024-04-13T03:32:41.907911+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:51:41.152066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'14","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731298+0000","last_change":"2024-04-13T03:31:36.598323+0000","last_active":"2024-04-13T03:32:41.731298+0000","last_peered":"2024-04-13T03:32:41.731298+0000","last_clean":"2024-04-13T03:32:41.731298+0000","last_became_active":"2024-04-13T03:31:36.597412+0000","last_became_peered":"2024-04-13T03:31:36.597412+0000","last_unstale":"2024-04-13T03:32:41.731298+0000","last_undegraded":"2024-04-13T03:32:41.731298+0000","last_fullsized":"2024-04-13T03:32:41.731298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:19:19.203273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.891938+0000","last_change":"2024-04-13T03:31:37.591599+0000","last_active":"2024-04-13T03:33:09.891938+0000","last_peered":"2024-04-13T03:33:09.891938+0000","last_clean":"2024-04-13T03:33:09.891938+0000","last_became_active":"2024-04-13T03:31:37.591243+0000","last_became_peered":"2024-04-13T03:31:37.591243+0000","last_unstale":"2024-04-13T03:33:09.891938+0000","last_undegraded":"2024-04-13T03:33:09.891938+0000","last_fullsized":"2024-04-13T03:33:09.891938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:11:35.460721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.736480+0000","last_change":"2024-04-13T03:31:41.821604+0000","last_active":"2024-04-13T03:32:41.736480+0000","last_peered":"2024-04-13T03:32:41.736480+0000","last_clean":"2024-04-13T03:32:41.736480+0000","last_became_active":"2024-04-13T03:31:41.820804+0000","last_became_peered":"2024-04-13T03:31:41.820804+0000","last_unstale":"2024-04-13T03:32:41.736480+0000","last_undegraded":"2024-04-13T03:32:41.736480+0000","last_fullsized":"2024-04-13T03:32:41.736480+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:10:45.850099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728904+0000","last_change":"2024-04-13T03:31:36.582624+0000","last_active":"2024-04-13T03:32:41.728904+0000","last_peered":"2024-04-13T03:32:41.728904+0000","last_clean":"2024-04-13T03:32:41.728904+0000","last_became_active":"2024-04-13T03:31:36.582260+0000","last_became_peered":"2024-04-13T03:31:36.582260+0000","last_unstale":"2024-04-13T03:32:41.728904+0000","last_undegraded":"2024-04-13T03:32:41.728904+0000","last_fullsized":"2024-04-13T03:32:41.728904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:15.637016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.891518+0000","last_change":"2024-04-13T03:31:37.591286+0000","last_active":"2024-04-13T03:33:09.891518+0000","last_peered":"2024-04-13T03:33:09.891518+0000","last_clean":"2024-04-13T03:33:09.891518+0000","last_became_active":"2024-04-13T03:31:37.590891+0000","last_became_peered":"2024-04-13T03:31:37.590891+0000","last_unstale":"2024-04-13T03:33:09.891518+0000","last_undegraded":"2024-04-13T03:33:09.891518+0000","last_fullsized":"2024-04-13T03:33:09.891518+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:53:54.696082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728853+0000","last_change":"2024-04-13T03:31:41.830499+0000","last_active":"2024-04-13T03:32:41.728853+0000","last_peered":"2024-04-13T03:32:41.728853+0000","last_clean":"2024-04-13T03:32:41.728853+0000","last_became_active":"2024-04-13T03:31:41.830067+0000","last_became_peered":"2024-04-13T03:31:41.830067+0000","last_unstale":"2024-04-13T03:32:41.728853+0000","last_undegraded":"2024-04-13T03:32:41.728853+0000","last_fullsized":"2024-04-13T03:32:41.728853+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:22.635885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.753157+0000","last_change":"2024-04-13T03:32:41.753157+0000","last_active":"2024-04-13T03:32:41.753157+0000","last_peered":"2024-04-13T03:32:41.753157+0000","last_clean":"2024-04-13T03:32:41.753157+0000","last_became_active":"2024-04-13T03:32:41.742206+0000","last_became_peered":"2024-04-13T03:32:41.742206+0000","last_unstale":"2024-04-13T03:32:41.753157+0000","last_undegraded":"2024-04-13T03:32:41.753157+0000","last_fullsized":"2024-04-13T03:32:41.753157+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:27:46.799187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.846009+0000","last_change":"2024-04-13T03:31:37.575749+0000","last_active":"2024-04-13T03:33:09.846009+0000","last_peered":"2024-04-13T03:33:09.846009+0000","last_clean":"2024-04-13T03:33:09.846009+0000","last_became_active":"2024-04-13T03:31:37.574152+0000","last_became_peered":"2024-04-13T03:31:37.574152+0000","last_unstale":"2024-04-13T03:33:09.846009+0000","last_undegraded":"2024-04-13T03:33:09.846009+0000","last_fullsized":"2024-04-13T03:33:09.846009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:38:15.710978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.731574+0000","last_change":"2024-04-13T03:31:41.836777+0000","last_active":"2024-04-13T03:32:41.731574+0000","last_peered":"2024-04-13T03:32:41.731574+0000","last_clean":"2024-04-13T03:32:41.731574+0000","last_became_active":"2024-04-13T03:31:41.835877+0000","last_became_peered":"2024-04-13T03:31:41.835877+0000","last_unstale":"2024-04-13T03:32:41.731574+0000","last_undegraded":"2024-04-13T03:32:41.731574+0000","last_fullsized":"2024-04-13T03:32:41.731574+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T10:46:44.174095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.909200+0000","last_change":"2024-04-13T03:31:36.587981+0000","last_active":"2024-04-13T03:32:41.909200+0000","last_peered":"2024-04-13T03:32:41.909200+0000","last_clean":"2024-04-13T03:32:41.909200+0000","last_became_active":"2024-04-13T03:31:36.586013+0000","last_became_peered":"2024-04-13T03:31:36.586013+0000","last_unstale":"2024-04-13T03:32:41.909200+0000","last_undegraded":"2024-04-13T03:32:41.909200+0000","last_fullsized":"2024-04-13T03:32:41.909200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:16:27.075322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.872088+0000","last_change":"2024-04-13T03:31:37.594185+0000","last_active":"2024-04-13T03:33:09.872088+0000","last_peered":"2024-04-13T03:33:09.872088+0000","last_clean":"2024-04-13T03:33:09.872088+0000","last_became_active":"2024-04-13T03:31:37.593791+0000","last_became_peered":"2024-04-13T03:31:37.593791+0000","last_unstale":"2024-04-13T03:33:09.872088+0000","last_undegraded":"2024-04-13T03:33:09.872088+0000","last_fullsized":"2024-04-13T03:33:09.872088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:43:32.035371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728904+0000","last_change":"2024-04-13T03:31:41.830392+0000","last_active":"2024-04-13T03:32:41.728904+0000","last_peered":"2024-04-13T03:32:41.728904+0000","last_clean":"2024-04-13T03:32:41.728904+0000","last_became_active":"2024-04-13T03:31:41.829967+0000","last_became_peered":"2024-04-13T03:31:41.829967+0000","last_unstale":"2024-04-13T03:32:41.728904+0000","last_undegraded":"2024-04-13T03:32:41.728904+0000","last_fullsized":"2024-04-13T03:32:41.728904+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:43:25.326951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.736877+0000","last_change":"2024-04-13T03:31:36.597099+0000","last_active":"2024-04-13T03:32:41.736877+0000","last_peered":"2024-04-13T03:32:41.736877+0000","last_clean":"2024-04-13T03:32:41.736877+0000","last_became_active":"2024-04-13T03:31:36.596714+0000","last_became_peered":"2024-04-13T03:31:36.596714+0000","last_unstale":"2024-04-13T03:32:41.736877+0000","last_undegraded":"2024-04-13T03:32:41.736877+0000","last_fullsized":"2024-04-13T03:32:41.736877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:42:31.477730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.897485+0000","last_change":"2024-04-13T03:31:37.587038+0000","last_active":"2024-04-13T03:33:09.897485+0000","last_peered":"2024-04-13T03:33:09.897485+0000","last_clean":"2024-04-13T03:33:09.897485+0000","last_became_active":"2024-04-13T03:31:37.586618+0000","last_became_peered":"2024-04-13T03:31:37.586618+0000","last_unstale":"2024-04-13T03:33:09.897485+0000","last_undegraded":"2024-04-13T03:33:09.897485+0000","last_fullsized":"2024-04-13T03:33:09.897485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:57:33.868529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.729704+0000","last_change":"2024-04-13T03:31:41.833148+0000","last_active":"2024-04-13T03:32:41.729704+0000","last_peered":"2024-04-13T03:32:41.729704+0000","last_clean":"2024-04-13T03:32:41.729704+0000","last_became_active":"2024-04-13T03:31:41.832714+0000","last_became_peered":"2024-04-13T03:31:41.832714+0000","last_unstale":"2024-04-13T03:32:41.729704+0000","last_undegraded":"2024-04-13T03:32:41.729704+0000","last_fullsized":"2024-04-13T03:32:41.729704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:12:07.316488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.728629+0000","last_change":"2024-04-13T03:31:36.564887+0000","last_active":"2024-04-13T03:32:41.728629+0000","last_peered":"2024-04-13T03:32:41.728629+0000","last_clean":"2024-04-13T03:32:41.728629+0000","last_became_active":"2024-04-13T03:31:36.564438+0000","last_became_peered":"2024-04-13T03:31:36.564438+0000","last_unstale":"2024-04-13T03:32:41.728629+0000","last_undegraded":"2024-04-13T03:32:41.728629+0000","last_fullsized":"2024-04-13T03:32:41.728629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:19.310322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.889488+0000","last_change":"2024-04-13T03:31:37.584358+0000","last_active":"2024-04-13T03:33:09.889488+0000","last_peered":"2024-04-13T03:33:09.889488+0000","last_clean":"2024-04-13T03:33:09.889488+0000","last_became_active":"2024-04-13T03:31:37.584005+0000","last_became_peered":"2024-04-13T03:31:37.584005+0000","last_unstale":"2024-04-13T03:33:09.889488+0000","last_undegraded":"2024-04-13T03:33:09.889488+0000","last_fullsized":"2024-04-13T03:33:09.889488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:07:21.465290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.733665+0000","last_change":"2024-04-13T03:31:41.833451+0000","last_active":"2024-04-13T03:32:41.733665+0000","last_peered":"2024-04-13T03:32:41.733665+0000","last_clean":"2024-04-13T03:32:41.733665+0000","last_became_active":"2024-04-13T03:31:41.832734+0000","last_became_peered":"2024-04-13T03:31:41.832734+0000","last_unstale":"2024-04-13T03:32:41.733665+0000","last_undegraded":"2024-04-13T03:32:41.733665+0000","last_fullsized":"2024-04-13T03:32:41.733665+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:49:48.848152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":123,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:34.938155+0000","last_change":"2024-04-13T03:31:36.581697+0000","last_active":"2024-04-13T03:33:34.938155+0000","last_peered":"2024-04-13T03:33:34.938155+0000","last_clean":"2024-04-13T03:33:34.938155+0000","last_became_active":"2024-04-13T03:31:36.581345+0000","last_became_peered":"2024-04-13T03:31:36.581345+0000","last_unstale":"2024-04-13T03:33:34.938155+0000","last_undegraded":"2024-04-13T03:33:34.938155+0000","last_fullsized":"2024-04-13T03:33:34.938155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:29:53.192770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887198+0000","last_change":"2024-04-13T03:31:37.571879+0000","last_active":"2024-04-13T03:33:09.887198+0000","last_peered":"2024-04-13T03:33:09.887198+0000","last_clean":"2024-04-13T03:33:09.887198+0000","last_became_active":"2024-04-13T03:31:37.571618+0000","last_became_peered":"2024-04-13T03:31:37.571618+0000","last_unstale":"2024-04-13T03:33:09.887198+0000","last_undegraded":"2024-04-13T03:33:09.887198+0000","last_fullsized":"2024-04-13T03:33:09.887198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:42.920388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.734871+0000","last_change":"2024-04-13T03:31:41.851871+0000","last_active":"2024-04-13T03:32:41.734871+0000","last_peered":"2024-04-13T03:32:41.734871+0000","last_clean":"2024-04-13T03:32:41.734871+0000","last_became_active":"2024-04-13T03:31:41.851045+0000","last_became_peered":"2024-04-13T03:31:41.851045+0000","last_unstale":"2024-04-13T03:32:41.734871+0000","last_undegraded":"2024-04-13T03:32:41.734871+0000","last_fullsized":"2024-04-13T03:32:41.734871+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:35:41.937723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.732348+0000","last_change":"2024-04-13T03:31:36.586110+0000","last_active":"2024-04-13T03:32:41.732348+0000","last_peered":"2024-04-13T03:32:41.732348+0000","last_clean":"2024-04-13T03:32:41.732348+0000","last_became_active":"2024-04-13T03:31:36.585832+0000","last_became_peered":"2024-04-13T03:31:36.585832+0000","last_unstale":"2024-04-13T03:32:41.732348+0000","last_undegraded":"2024-04-13T03:32:41.732348+0000","last_fullsized":"2024-04-13T03:32:41.732348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:00:57.825760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.884320+0000","last_change":"2024-04-13T03:31:37.580004+0000","last_active":"2024-04-13T03:33:09.884320+0000","last_peered":"2024-04-13T03:33:09.884320+0000","last_clean":"2024-04-13T03:33:09.884320+0000","last_became_active":"2024-04-13T03:31:37.579634+0000","last_became_peered":"2024-04-13T03:31:37.579634+0000","last_unstale":"2024-04-13T03:33:09.884320+0000","last_undegraded":"2024-04-13T03:33:09.884320+0000","last_fullsized":"2024-04-13T03:33:09.884320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:39:59.872588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.733211+0000","last_change":"2024-04-13T03:31:41.825456+0000","last_active":"2024-04-13T03:32:41.733211+0000","last_peered":"2024-04-13T03:32:41.733211+0000","last_clean":"2024-04-13T03:32:41.733211+0000","last_became_active":"2024-04-13T03:31:41.824929+0000","last_became_peered":"2024-04-13T03:31:41.824929+0000","last_unstale":"2024-04-13T03:32:41.733211+0000","last_undegraded":"2024-04-13T03:32:41.733211+0000","last_fullsized":"2024-04-13T03:32:41.733211+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:40.799093+0000","last_clean_scrub_stamp":"2024-04-13T03:31:40.799093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:17:39.405147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:32:41.741969+0000","last_change":"2024-04-13T03:31:36.595009+0000","last_active":"2024-04-13T03:32:41.741969+0000","last_peered":"2024-04-13T03:32:41.741969+0000","last_clean":"2024-04-13T03:32:41.741969+0000","last_became_active":"2024-04-13T03:31:36.594314+0000","last_became_peered":"2024-04-13T03:31:36.594314+0000","last_unstale":"2024-04-13T03:32:41.741969+0000","last_undegraded":"2024-04-13T03:32:41.741969+0000","last_fullsized":"2024-04-13T03:32:41.741969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:35.525151+0000","last_clean_scrub_stamp":"2024-04-13T03:31:35.525151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:42:33.570088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.878633+0000","last_change":"2024-04-13T03:31:37.579899+0000","last_active":"2024-04-13T03:33:09.878633+0000","last_peered":"2024-04-13T03:33:09.878633+0000","last_clean":"2024-04-13T03:33:09.878633+0000","last_became_active":"2024-04-13T03:31:37.579345+0000","last_became_peered":"2024-04-13T03:31:37.579345+0000","last_unstale":"2024-04-13T03:33:09.878633+0000","last_undegraded":"2024-04-13T03:33:09.878633+0000","last_fullsized":"2024-04-13T03:33:09.878633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:06:55.646866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.868497+0000","last_change":"2024-04-13T03:31:37.586373+0000","last_active":"2024-04-13T03:33:09.868497+0000","last_peered":"2024-04-13T03:33:09.868497+0000","last_clean":"2024-04-13T03:33:09.868497+0000","last_became_active":"2024-04-13T03:31:37.586061+0000","last_became_peered":"2024-04-13T03:31:37.586061+0000","last_unstale":"2024-04-13T03:33:09.868497+0000","last_undegraded":"2024-04-13T03:33:09.868497+0000","last_fullsized":"2024-04-13T03:33:09.868497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T13:58:52.487187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.866958+0000","last_change":"2024-04-13T03:31:37.581375+0000","last_active":"2024-04-13T03:33:09.866958+0000","last_peered":"2024-04-13T03:33:09.866958+0000","last_clean":"2024-04-13T03:33:09.866958+0000","last_became_active":"2024-04-13T03:31:37.580802+0000","last_became_peered":"2024-04-13T03:31:37.580802+0000","last_unstale":"2024-04-13T03:33:09.866958+0000","last_undegraded":"2024-04-13T03:33:09.866958+0000","last_fullsized":"2024-04-13T03:33:09.866958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:41.895243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.892179+0000","last_change":"2024-04-13T03:31:37.580598+0000","last_active":"2024-04-13T03:33:09.892179+0000","last_peered":"2024-04-13T03:33:09.892179+0000","last_clean":"2024-04-13T03:33:09.892179+0000","last_became_active":"2024-04-13T03:31:37.580276+0000","last_became_peered":"2024-04-13T03:31:37.580276+0000","last_unstale":"2024-04-13T03:33:09.892179+0000","last_undegraded":"2024-04-13T03:33:09.892179+0000","last_fullsized":"2024-04-13T03:33:09.892179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:56:02.302300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.895650+0000","last_change":"2024-04-13T03:31:37.596435+0000","last_active":"2024-04-13T03:33:09.895650+0000","last_peered":"2024-04-13T03:33:09.895650+0000","last_clean":"2024-04-13T03:33:09.895650+0000","last_became_active":"2024-04-13T03:31:37.596095+0000","last_became_peered":"2024-04-13T03:31:37.596095+0000","last_unstale":"2024-04-13T03:33:09.895650+0000","last_undegraded":"2024-04-13T03:33:09.895650+0000","last_fullsized":"2024-04-13T03:33:09.895650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:47:04.425037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.895926+0000","last_change":"2024-04-13T03:31:37.578003+0000","last_active":"2024-04-13T03:33:09.895926+0000","last_peered":"2024-04-13T03:33:09.895926+0000","last_clean":"2024-04-13T03:33:09.895926+0000","last_became_active":"2024-04-13T03:31:37.577637+0000","last_became_peered":"2024-04-13T03:31:37.577637+0000","last_unstale":"2024-04-13T03:33:09.895926+0000","last_undegraded":"2024-04-13T03:33:09.895926+0000","last_fullsized":"2024-04-13T03:33:09.895926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:30:37.138611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.864965+0000","last_change":"2024-04-13T03:31:37.585404+0000","last_active":"2024-04-13T03:33:09.864965+0000","last_peered":"2024-04-13T03:33:09.864965+0000","last_clean":"2024-04-13T03:33:09.864965+0000","last_became_active":"2024-04-13T03:31:37.585060+0000","last_became_peered":"2024-04-13T03:31:37.585060+0000","last_unstale":"2024-04-13T03:33:09.864965+0000","last_undegraded":"2024-04-13T03:33:09.864965+0000","last_fullsized":"2024-04-13T03:33:09.864965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T14:37:52.165706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.865243+0000","last_change":"2024-04-13T03:31:37.576265+0000","last_active":"2024-04-13T03:33:09.865243+0000","last_peered":"2024-04-13T03:33:09.865243+0000","last_clean":"2024-04-13T03:33:09.865243+0000","last_became_active":"2024-04-13T03:31:37.575906+0000","last_became_peered":"2024-04-13T03:31:37.575906+0000","last_unstale":"2024-04-13T03:33:09.865243+0000","last_undegraded":"2024-04-13T03:33:09.865243+0000","last_fullsized":"2024-04-13T03:33:09.865243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T11:31:34.505830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.893424+0000","last_change":"2024-04-13T03:31:37.582865+0000","last_active":"2024-04-13T03:33:09.893424+0000","last_peered":"2024-04-13T03:33:09.893424+0000","last_clean":"2024-04-13T03:33:09.893424+0000","last_became_active":"2024-04-13T03:31:37.581055+0000","last_became_peered":"2024-04-13T03:31:37.581055+0000","last_unstale":"2024-04-13T03:33:09.893424+0000","last_undegraded":"2024-04-13T03:33:09.893424+0000","last_fullsized":"2024-04-13T03:33:09.893424+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:01:07.815483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.893615+0000","last_change":"2024-04-13T03:31:37.587677+0000","last_active":"2024-04-13T03:33:09.893615+0000","last_peered":"2024-04-13T03:33:09.893615+0000","last_clean":"2024-04-13T03:33:09.893615+0000","last_became_active":"2024-04-13T03:31:37.586778+0000","last_became_peered":"2024-04-13T03:31:37.586778+0000","last_unstale":"2024-04-13T03:33:09.893615+0000","last_undegraded":"2024-04-13T03:33:09.893615+0000","last_fullsized":"2024-04-13T03:33:09.893615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T12:52:13.968559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.894682+0000","last_change":"2024-04-13T03:31:37.577694+0000","last_active":"2024-04-13T03:33:09.894682+0000","last_peered":"2024-04-13T03:33:09.894682+0000","last_clean":"2024-04-13T03:33:09.894682+0000","last_became_active":"2024-04-13T03:31:37.577287+0000","last_became_peered":"2024-04-13T03:31:37.577287+0000","last_unstale":"2024-04-13T03:33:09.894682+0000","last_undegraded":"2024-04-13T03:33:09.894682+0000","last_fullsized":"2024-04-13T03:33:09.894682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:16:26.503330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.887902+0000","last_change":"2024-04-13T03:31:37.591160+0000","last_active":"2024-04-13T03:33:09.887902+0000","last_peered":"2024-04-13T03:33:09.887902+0000","last_clean":"2024-04-13T03:33:09.887902+0000","last_became_active":"2024-04-13T03:31:37.590666+0000","last_became_peered":"2024-04-13T03:31:37.590666+0000","last_unstale":"2024-04-13T03:33:09.887902+0000","last_undegraded":"2024-04-13T03:33:09.887902+0000","last_fullsized":"2024-04-13T03:33:09.887902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:43:25.976108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.896823+0000","last_change":"2024-04-13T03:31:37.594433+0000","last_active":"2024-04-13T03:33:09.896823+0000","last_peered":"2024-04-13T03:33:09.896823+0000","last_clean":"2024-04-13T03:33:09.896823+0000","last_became_active":"2024-04-13T03:31:37.590942+0000","last_became_peered":"2024-04-13T03:31:37.590942+0000","last_unstale":"2024-04-13T03:33:09.896823+0000","last_undegraded":"2024-04-13T03:33:09.896823+0000","last_fullsized":"2024-04-13T03:33:09.896823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:42:47.534702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-13T03:33:09.869884+0000","last_change":"2024-04-13T03:31:37.583342+0000","last_active":"2024-04-13T03:33:09.869884+0000","last_peered":"2024-04-13T03:33:09.869884+0000","last_clean":"2024-04-13T03:33:09.869884+0000","last_became_active":"2024-04-13T03:31:37.582983+0000","last_became_peered":"2024-04-13T03:31:37.582983+0000","last_unstale":"2024-04-13T03:33:09.869884+0000","last_undegraded":"2024-04-13T03:33:09.869884+0000","last_fullsized":"2024-04-13T03:33:09.869884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-13T03:31:36.535123+0000","last_clean_scrub_stamp":"2024-04-13T03:31:36.535123+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:24:26.616843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284330,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71941,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":914,"num_read_kb":533,"num_write":253,"num_write_kb":149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":64318,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":215823,"data_compressed":14002,"data_compressed_allocated":20480,"data_compressed_original":77824,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351014,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":508208,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612651,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1159899,"data_compressed":58649,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":512122,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":508292,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1105574,"data_compressed":56822,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":503530,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":569015,"data_compressed":22492,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":504986,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":505312,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":503238,"data_compressed":20665,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":572591,"data_compressed":31013,"data_compressed_allocated":40960,"data_compressed_original":118784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":1100624,"data_compressed":56822,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34515,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36157,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66207,"data_compressed":1827,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":65687,"data_compressed":10348,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":66723,"data_compressed":1827,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-13T03:33:39.494 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-13T03:33:39.494 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-13T03:33:39.494 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-13T03:33:39.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph health --format=json 2024-04-13T03:33:39.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:39 smithi119 ceph-mon[35835]: pgmap v381: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:39.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:39 smithi119 ceph-mon[35835]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:33:39.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:39 smithi188 ceph-mon[36398]: pgmap v381: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:33:39.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:39 smithi188 ceph-mon[36398]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-13T03:33:40.004 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:41.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:41 smithi119 ceph-mon[35835]: pgmap v382: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:33:41.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-13T03:33:41.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-13T03:33:41.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-04-13T03:33:41.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:41 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]: dispatch 2024-04-13T03:33:41.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:41 smithi016 ceph-mon[31772]: pgmap v382: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:33:41.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-13T03:33:41.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-13T03:33:41.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-04-13T03:33:41.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:41 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]: dispatch 2024-04-13T03:33:41.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:41 smithi188 ceph-mon[36398]: pgmap v382: 129 pgs: 129 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:33:41.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-13T03:33:41.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-13T03:33:41.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-04-13T03:33:41.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:41 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]: dispatch 2024-04-13T03:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.433+0000 7f7874600700 1 -- 172.21.15.16:0/1424063529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c075600 msgr2=0x7f786c10efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.433+0000 7f7874600700 1 --2- 172.21.15.16:0/1424063529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c075600 0x7f786c10efa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7858009990 tx=0x7f7858009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.433+0000 7f7874600700 1 -- 172.21.15.16:0/1424063529 shutdown_connections 2024-04-13T03:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.433+0000 7f7874600700 1 --2- 172.21.15.16:0/1424063529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f786c10f4e0 0x7f786c1118d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.433+0000 7f7874600700 1 --2- 172.21.15.16:0/1424063529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f786c074ce0 0x7f786c0750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.433+0000 7f7874600700 1 --2- 172.21.15.16:0/1424063529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c075600 0x7f786c10efa0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.433+0000 7f7874600700 1 -- 172.21.15.16:0/1424063529 >> 172.21.15.16:0/1424063529 conn(0x7f786c079b80 msgr2=0x7f786c079f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.433+0000 7f7874600700 1 -- 172.21.15.16:0/1424063529 shutdown_connections 2024-04-13T03:33:42.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.433+0000 7f7874600700 1 -- 172.21.15.16:0/1424063529 wait complete. 2024-04-13T03:33:42.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-13T03:33:42.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-13T03:33:42.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-04-13T03:33:42.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:42 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]': finished 2024-04-13T03:33:42.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:42 smithi016 ceph-mon[31772]: osdmap e93: 12 total, 12 up, 12 in 2024-04-13T03:33:42.436 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:42 smithi016 ceph-mon[31772]: osdmap e94: 12 total, 12 up, 12 in 2024-04-13T03:33:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.434+0000 7f7874600700 1 Processor -- start 2024-04-13T03:33:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.434+0000 7f7874600700 1 -- start start 2024-04-13T03:33:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7874600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c074ce0 0x7f786c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7874600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f786c075600 0x7f786c19b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7874600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f786c10f4e0 0x7f786c19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:42.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7874600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f786c115010 con 0x7f786c10f4e0 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7874600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f786c114e90 con 0x7f786c075600 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7874600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f786c115190 con 0x7f786c074ce0 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7872600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f786c10f4e0 0x7f786c19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7871200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f786c075600 0x7f786c19b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7871c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c074ce0 0x7f786c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7871200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f786c075600 0x7f786c19b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:35640/0 (socket says 172.21.15.16:35640) 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7871c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c074ce0 0x7f786c19b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:35198/0 (socket says 172.21.15.16:35198) 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.435+0000 7f7871200700 1 -- 172.21.15.16:0/788134637 learned_addr learned my addr 172.21.15.16:0/788134637 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.436+0000 7f7871c00700 1 -- 172.21.15.16:0/788134637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f786c075600 msgr2=0x7f786c19b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.436+0000 7f7871c00700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f786c075600 0x7f786c19b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.436+0000 7f7871c00700 1 -- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f786c10f4e0 msgr2=0x7f786c19fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.436+0000 7f7871c00700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f786c10f4e0 0x7f786c19fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.436+0000 7f7871c00700 1 -- 172.21.15.16:0/788134637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c1a03e0 con 0x7f786c074ce0 2024-04-13T03:33:42.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.436+0000 7f7872600700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f786c10f4e0 0x7f786c19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:42.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.436+0000 7f7871c00700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c074ce0 0x7f786c19b420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f786000e9f0 tx=0x7f786000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:42.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.437+0000 7f786a000700 1 -- 172.21.15.16:0/788134637 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786000c9c0 con 0x7f786c074ce0 2024-04-13T03:33:42.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.437+0000 7f786a000700 1 -- 172.21.15.16:0/788134637 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786000cb60 con 0x7f786c074ce0 2024-04-13T03:33:42.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.437+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786c1a06d0 con 0x7f786c074ce0 2024-04-13T03:33:42.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.437+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786c1a7f50 con 0x7f786c074ce0 2024-04-13T03:33:42.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.437+0000 7f786a000700 1 -- 172.21.15.16:0/788134637 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7860018680 con 0x7f786c074ce0 2024-04-13T03:33:42.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.438+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f784c005330 con 0x7f786c074ce0 2024-04-13T03:33:42.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.440+0000 7f786a000700 1 -- 172.21.15.16:0/788134637 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f786000cd10 con 0x7f786c074ce0 2024-04-13T03:33:42.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.441+0000 7f786a000700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f78540a3c40 0x7f78540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:42.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.442+0000 7f786a000700 1 -- 172.21.15.16:0/788134637 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f78600d0860 con 0x7f786c074ce0 2024-04-13T03:33:42.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.442+0000 7f7871200700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f78540a3c40 0x7f78540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:42.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.442+0000 7f7871200700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f78540a3c40 0x7f78540a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7858009750 tx=0x7f7858005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:42.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.446+0000 7f786a000700 1 -- 172.21.15.16:0/788134637 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f786009ce10 con 0x7f786c074ce0 2024-04-13T03:33:42.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-13T03:33:42.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-13T03:33:42.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-04-13T03:33:42.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:42 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]': finished 2024-04-13T03:33:42.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:42 smithi119 ceph-mon[35835]: osdmap e93: 12 total, 12 up, 12 in 2024-04-13T03:33:42.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:42 smithi119 ceph-mon[35835]: osdmap e94: 12 total, 12 up, 12 in 2024-04-13T03:33:42.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-13T03:33:42.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-13T03:33:42.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-04-13T03:33:42.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:42 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]': finished 2024-04-13T03:33:42.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:42 smithi188 ceph-mon[36398]: osdmap e93: 12 total, 12 up, 12 in 2024-04-13T03:33:42.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:42 smithi188 ceph-mon[36398]: osdmap e94: 12 total, 12 up, 12 in 2024-04-13T03:33:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.729+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f784c005620 con 0x7f786c074ce0 2024-04-13T03:33:42.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.730+0000 7f786a000700 1 -- 172.21.15.16:0/788134637 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+181 (secure 0 0 0) 0x7f786009c710 con 0x7f786c074ce0 2024-04-13T03:33:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:42.731 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 2 pgs peering","count":2},"muted":false}},"mutes":[]} 2024-04-13T03:33:42.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f78540a3c40 msgr2=0x7f78540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:42.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f78540a3c40 0x7f78540a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7858009750 tx=0x7f7858005960 comp rx=0 tx=0).stop 2024-04-13T03:33:42.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c074ce0 msgr2=0x7f786c19b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:42.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c074ce0 0x7f786c19b420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f786000e9f0 tx=0x7f786000ed00 comp rx=0 tx=0).stop 2024-04-13T03:33:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 shutdown_connections 2024-04-13T03:33:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f78540a3c40 0x7f78540a6100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f786c10f4e0 0x7f786c19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f786c074ce0 0x7f786c19b420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 --2- 172.21.15.16:0/788134637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f786c075600 0x7f786c19b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.734+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 >> 172.21.15.16:0/788134637 conn(0x7f786c079b80 msgr2=0x7f786c0ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.735+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 shutdown_connections 2024-04-13T03:33:42.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:42.735+0000 7f7874600700 1 -- 172.21.15.16:0/788134637 wait complete. 2024-04-13T03:33:43.459 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:43 smithi016 ceph-mon[31772]: pgmap v384: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-13T03:33:43.459 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:43 smithi016 ceph-mon[31772]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-13T03:33:43.459 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:43 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/788134637' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-13T03:33:43.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:43 smithi119 ceph-mon[35835]: pgmap v384: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-13T03:33:43.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:43 smithi119 ceph-mon[35835]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-13T03:33:43.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:43 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/788134637' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-13T03:33:43.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:43 smithi188 ceph-mon[36398]: pgmap v384: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-13T03:33:43.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:43 smithi188 ceph-mon[36398]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-13T03:33:43.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:43 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/788134637' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-13T03:33:45.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:45 smithi119 ceph-mon[35835]: pgmap v386: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-13T03:33:45.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:45 smithi016 ceph-mon[31772]: pgmap v386: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-13T03:33:45.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:45 smithi188 ceph-mon[36398]: pgmap v386: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-04-13T03:33:46.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph health --format=json 2024-04-13T03:33:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:47.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:47 smithi016 ceph-mon[31772]: pgmap v387: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-04-13T03:33:47.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:47 smithi119 ceph-mon[35835]: pgmap v387: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-04-13T03:33:47.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:47 smithi188 ceph-mon[36398]: pgmap v387: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-04-13T03:33:48.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:48 smithi016 ceph-mon[31772]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-13T03:33:48.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:48 smithi016 ceph-mon[31772]: Cluster is now healthy 2024-04-13T03:33:48.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:48 smithi119 ceph-mon[35835]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-13T03:33:48.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:48 smithi119 ceph-mon[35835]: Cluster is now healthy 2024-04-13T03:33:48.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:48 smithi188 ceph-mon[36398]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-13T03:33:48.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:48 smithi188 ceph-mon[36398]: Cluster is now healthy 2024-04-13T03:33:49.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.253+0000 7fb2ba400700 1 -- 172.21.15.16:0/1670053940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2b406b050 msgr2=0x7fb2b406b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:49.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.253+0000 7fb2ba400700 1 --2- 172.21.15.16:0/1670053940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2b406b050 0x7fb2b406b430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009990 tx=0x7fb2a4009ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.253+0000 7fb2ba400700 1 -- 172.21.15.16:0/1670053940 shutdown_connections 2024-04-13T03:33:49.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.253+0000 7fb2ba400700 1 --2- 172.21.15.16:0/1670053940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2b410f550 0x7fb2b4111940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.253+0000 7fb2ba400700 1 --2- 172.21.15.16:0/1670053940 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2b406b050 0x7fb2b406b430 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.253+0000 7fb2ba400700 1 --2- 172.21.15.16:0/1670053940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2b406b970 0x7fb2b410f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.253+0000 7fb2ba400700 1 -- 172.21.15.16:0/1670053940 >> 172.21.15.16:0/1670053940 conn(0x7fb2b4078920 msgr2=0x7fb2b4078d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:49.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.254+0000 7fb2ba400700 1 -- 172.21.15.16:0/1670053940 shutdown_connections 2024-04-13T03:33:49.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.254+0000 7fb2ba400700 1 -- 172.21.15.16:0/1670053940 wait complete. 2024-04-13T03:33:49.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.255+0000 7fb2ba400700 1 Processor -- start 2024-04-13T03:33:49.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.255+0000 7fb2ba400700 1 -- start start 2024-04-13T03:33:49.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2ba400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2b406b050 0x7fb2b419b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:49.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2ba400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2b406b970 0x7fb2b419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:49.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2ba400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2b410f550 0x7fb2b419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:49.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2ba400700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b4114fb0 con 0x7fb2b406b970 2024-04-13T03:33:49.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2ba400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b4114e30 con 0x7fb2b410f550 2024-04-13T03:33:49.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2ba400700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b4115130 con 0x7fb2b406b050 2024-04-13T03:33:49.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2b406b050 0x7fb2b419b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:49.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2b3400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2b406b050 0x7fb2b419b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53034/0 (socket says 172.21.15.16:53034) 2024-04-13T03:33:49.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2b3400700 1 -- 172.21.15.16:0/4207180739 learned_addr learned my addr 172.21.15.16:0/4207180739 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:49.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2b2a00700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2b406b970 0x7fb2b419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:49.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.256+0000 7fb2b3e00700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2b410f550 0x7fb2b419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:49.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.257+0000 7fb2b3400700 1 -- 172.21.15.16:0/4207180739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2b410f550 msgr2=0x7fb2b419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:49.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.257+0000 7fb2b3400700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2b410f550 0x7fb2b419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.257+0000 7fb2b3400700 1 -- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2b406b970 msgr2=0x7fb2b419ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:49.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.257+0000 7fb2b3400700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2b406b970 0x7fb2b419ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.257+0000 7fb2b3400700 1 -- 172.21.15.16:0/4207180739 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b41a0510 con 0x7fb2b406b050 2024-04-13T03:33:49.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.257+0000 7fb2b3400700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2b406b050 0x7fb2b419b500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4000c00 tx=0x7fb2a400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:49.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.257+0000 7fb2abe00700 1 -- 172.21.15.16:0/4207180739 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4027070 con 0x7fb2b406b050 2024-04-13T03:33:49.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.257+0000 7fb2abe00700 1 -- 172.21.15.16:0/4207180739 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2a4003d70 con 0x7fb2b406b050 2024-04-13T03:33:49.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.258+0000 7fb2abe00700 1 -- 172.21.15.16:0/4207180739 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a400f460 con 0x7fb2b406b050 2024-04-13T03:33:49.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.258+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b41a07a0 con 0x7fb2b406b050 2024-04-13T03:33:49.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.258+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b41a8080 con 0x7fb2b406b050 2024-04-13T03:33:49.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.259+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2b4101fc0 con 0x7fb2b406b050 2024-04-13T03:33:49.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.261+0000 7fb2abe00700 1 -- 172.21.15.16:0/4207180739 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb2a402a070 con 0x7fb2b406b050 2024-04-13T03:33:49.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.262+0000 7fb2abe00700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2980a3c40 0x7fb2980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:49.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.263+0000 7fb2abe00700 1 -- 172.21.15.16:0/4207180739 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb2a40d1780 con 0x7fb2b406b050 2024-04-13T03:33:49.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.263+0000 7fb2b2a00700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2980a3c40 0x7fb2980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:49.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.263+0000 7fb2b2a00700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2980a3c40 0x7fb2980a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4102f60 tx=0x7fb2a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:49.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.265+0000 7fb2abe00700 1 -- 172.21.15.16:0/4207180739 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2a4013070 con 0x7fb2b406b050 2024-04-13T03:33:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:49 smithi016 ceph-mon[31772]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-04-13T03:33:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.547+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb2b419c860 con 0x7fb2b406b050 2024-04-13T03:33:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.548+0000 7fb2abe00700 1 -- 172.21.15.16:0/4207180739 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb2a409ddc0 con 0x7fb2b406b050 2024-04-13T03:33:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:49.549 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-13T03:33:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2980a3c40 msgr2=0x7fb2980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2980a3c40 0x7fb2980a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4102f60 tx=0x7fb2a0009260 comp rx=0 tx=0).stop 2024-04-13T03:33:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2b406b050 msgr2=0x7fb2b419b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2b406b050 0x7fb2b419b500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4000c00 tx=0x7fb2a400bab0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 shutdown_connections 2024-04-13T03:33:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb2980a3c40 0x7fb2980a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2b410f550 0x7fb2b419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2b406b050 0x7fb2b419b500 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 --2- 172.21.15.16:0/4207180739 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2b406b970 0x7fb2b419ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 >> 172.21.15.16:0/4207180739 conn(0x7fb2b4078920 msgr2=0x7fb2b4110aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 shutdown_connections 2024-04-13T03:33:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:49.551+0000 7fb2ba400700 1 -- 172.21.15.16:0/4207180739 wait complete. 2024-04-13T03:33:49.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:49 smithi119 ceph-mon[35835]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-04-13T03:33:49.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:49 smithi188 ceph-mon[36398]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-04-13T03:33:50.421 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-13T03:33:50.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:33:50.516 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:50 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4207180739' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-13T03:33:50.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:50 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4207180739' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-13T03:33:50.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:50 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4207180739' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-13T03:33:50.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.662+0000 7fafe2600640 1 -- 172.21.15.16:0/448077375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafdc105230 msgr2=0x7fafdc1056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:50.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.663+0000 7fafe2600640 1 --2- 172.21.15.16:0/448077375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafdc105230 0x7fafdc1056b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009900 tx=0x7fafc802bc30 comp rx=0 tx=0).stop 2024-04-13T03:33:50.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.663+0000 7fafe2600640 1 -- 172.21.15.16:0/448077375 shutdown_connections 2024-04-13T03:33:50.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.663+0000 7fafe2600640 1 --2- 172.21.15.16:0/448077375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafdc105bf0 0x7fafdc10c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.663+0000 7fafe2600640 1 --2- 172.21.15.16:0/448077375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafdc105230 0x7fafdc1056b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.663+0000 7fafe2600640 1 --2- 172.21.15.16:0/448077375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafdc104040 0x7fafdc104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.663+0000 7fafe2600640 1 -- 172.21.15.16:0/448077375 >> 172.21.15.16:0/448077375 conn(0x7fafdc0ff5d0 msgr2=0x7fafdc1019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:50.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.663+0000 7fafe2600640 1 -- 172.21.15.16:0/448077375 shutdown_connections 2024-04-13T03:33:50.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.663+0000 7fafe2600640 1 -- 172.21.15.16:0/448077375 wait complete. 2024-04-13T03:33:50.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.664+0000 7fafe2600640 1 Processor -- start 2024-04-13T03:33:50.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.664+0000 7fafe2600640 1 -- start start 2024-04-13T03:33:50.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe2600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafdc104040 0x7fafdc19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:50.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe2600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafdc105230 0x7fafdc19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe2600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafdc105bf0 0x7fafdc1a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe2600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10fd20 con 0x7fafdc104040 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe2600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10fba0 con 0x7fafdc105bf0 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe2600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10fea0 con 0x7fafdc105230 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe1200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafdc104040 0x7fafdc19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe1c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafdc105bf0 0x7fafdc1a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe1200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafdc104040 0x7fafdc19a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59254/0 (socket says 172.21.15.16:59254) 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafe1200640 1 -- 172.21.15.16:0/3231386997 learned_addr learned my addr 172.21.15.16:0/3231386997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.665+0000 7fafdbe00640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafdc105230 0x7fafdc19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafe1c00640 1 -- 172.21.15.16:0/3231386997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafdc105230 msgr2=0x7fafdc19a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafe1c00640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafdc105230 0x7fafdc19a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafe1c00640 1 -- 172.21.15.16:0/3231386997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafdc104040 msgr2=0x7fafdc19a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafe1c00640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafdc104040 0x7fafdc19a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafe1c00640 1 -- 172.21.15.16:0/3231386997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafdc1a1f50 con 0x7fafdc105bf0 2024-04-13T03:33:50.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafe1c00640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafdc105bf0 0x7fafdc1a18e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fafd400ec50 tx=0x7fafd400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:50.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafd9600640 1 -- 172.21.15.16:0/3231386997 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd4003b70 con 0x7fafdc105bf0 2024-04-13T03:33:50.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafd9600640 1 -- 172.21.15.16:0/3231386997 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd4003d10 con 0x7fafdc105bf0 2024-04-13T03:33:50.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafdc1a2240 con 0x7fafdc105bf0 2024-04-13T03:33:50.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafd9600640 1 -- 172.21.15.16:0/3231386997 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd4010890 con 0x7fafdc105bf0 2024-04-13T03:33:50.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.666+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafdc1a2710 con 0x7fafdc105bf0 2024-04-13T03:33:50.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.668+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafdc10b360 con 0x7fafdc105bf0 2024-04-13T03:33:50.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.669+0000 7fafd9600640 1 -- 172.21.15.16:0/3231386997 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fafd4003fc0 con 0x7fafdc105bf0 2024-04-13T03:33:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.671+0000 7fafd9600640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fafc40a3bb0 0x7fafc40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.671+0000 7fafd9600640 1 -- 172.21.15.16:0/3231386997 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fafd40d0db0 con 0x7fafdc105bf0 2024-04-13T03:33:50.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.671+0000 7fafe1200640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fafc40a3bb0 0x7fafc40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:50.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.672+0000 7fafe1200640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fafc40a3bb0 0x7fafc40a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fafd00058c0 tx=0x7fafd0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:50.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.675+0000 7fafd9600640 1 -- 172.21.15.16:0/3231386997 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafd40d5070 con 0x7fafdc105bf0 2024-04-13T03:33:50.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.884+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fafdc062bc0 con 0x7fafdc105bf0 2024-04-13T03:33:50.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.886+0000 7fafd9600640 1 -- 172.21.15.16:0/3231386997 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24874 (secure 0 0 0) 0x7fafd40bbe20 con 0x7fafdc105bf0 2024-04-13T03:33:50.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:50.888 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":94,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:33:42.153326+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.888+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fafc40a3bb0 msgr2=0x7fafc40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.888+0000 7fafe2600640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fafc40a3bb0 0x7fafc40a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fafd00058c0 tx=0x7fafd0005f90 comp rx=0 tx=0).stop 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.888+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafdc105bf0 msgr2=0x7fafdc1a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.888+0000 7fafe2600640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafdc105bf0 0x7fafdc1a18e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fafd400ec50 tx=0x7fafd400c6a0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.889+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 shutdown_connections 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.889+0000 7fafe2600640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fafc40a3bb0 0x7fafc40a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.889+0000 7fafe2600640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafdc105bf0 0x7fafdc1a18e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.889+0000 7fafe2600640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafdc105230 0x7fafdc19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.889+0000 7fafe2600640 1 --2- 172.21.15.16:0/3231386997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafdc104040 0x7fafdc19a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:50.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.889+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 >> 172.21.15.16:0/3231386997 conn(0x7fafdc0ff5d0 msgr2=0x7fafdc108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.889+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 shutdown_connections 2024-04-13T03:33:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:50.889+0000 7fafe2600640 1 -- 172.21.15.16:0/3231386997 wait complete. 2024-04-13T03:33:50.910 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-13T03:31:34.802688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-13T03:31:36.036786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-13T03:31:39.824413+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-13T03:33:50.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:33:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.141+0000 7fa2db200640 1 -- 172.21.15.16:0/1243311856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2d4105410 msgr2=0x7fa2d4105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.141+0000 7fa2db200640 1 --2- 172.21.15.16:0/1243311856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2d4105410 0x7fa2d4105890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc009900 tx=0x7fa2bc02bc30 comp rx=0 tx=0).stop 2024-04-13T03:33:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.142+0000 7fa2db200640 1 -- 172.21.15.16:0/1243311856 shutdown_connections 2024-04-13T03:33:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.142+0000 7fa2db200640 1 --2- 172.21.15.16:0/1243311856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4105dd0 0x7fa2d410c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.142+0000 7fa2db200640 1 --2- 172.21.15.16:0/1243311856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2d4105410 0x7fa2d4105890 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.142+0000 7fa2db200640 1 --2- 172.21.15.16:0/1243311856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d4104220 0x7fa2d4104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.142+0000 7fa2db200640 1 -- 172.21.15.16:0/1243311856 >> 172.21.15.16:0/1243311856 conn(0x7fa2d40ff7b0 msgr2=0x7fa2d4101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.142+0000 7fa2db200640 1 -- 172.21.15.16:0/1243311856 shutdown_connections 2024-04-13T03:33:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.142+0000 7fa2db200640 1 -- 172.21.15.16:0/1243311856 wait complete. 2024-04-13T03:33:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.143+0000 7fa2db200640 1 Processor -- start 2024-04-13T03:33:51.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.143+0000 7fa2db200640 1 -- start start 2024-04-13T03:33:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2db200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2d4104220 0x7fa2d419a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2db200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4105410 0x7fa2d419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2db200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d4105dd0 0x7fa2d41a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:51.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2db200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d410ff00 con 0x7fa2d4105dd0 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2db200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d410fd80 con 0x7fa2d4104220 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2db200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d4110080 con 0x7fa2d4105410 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2cbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4105410 0x7fa2d419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2d9200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d4105dd0 0x7fa2d41a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2cbe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4105410 0x7fa2d419aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53068/0 (socket says 172.21.15.16:53068) 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2cbe00640 1 -- 172.21.15.16:0/2068693004 learned_addr learned my addr 172.21.15.16:0/2068693004 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2d3e00640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2d4104220 0x7fa2d419a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2cbe00640 1 -- 172.21.15.16:0/2068693004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2d4104220 msgr2=0x7fa2d419a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2cbe00640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2d4104220 0x7fa2d419a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2cbe00640 1 -- 172.21.15.16:0/2068693004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d4105dd0 msgr2=0x7fa2d41a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2cbe00640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d4105dd0 0x7fa2d41a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.144+0000 7fa2cbe00640 1 -- 172.21.15.16:0/2068693004 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d41a2130 con 0x7fa2d4105410 2024-04-13T03:33:51.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.145+0000 7fa2cbe00640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4105410 0x7fa2d419aa50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:51.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.145+0000 7fa2d1600640 1 -- 172.21.15.16:0/2068693004 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc043020 con 0x7fa2d4105410 2024-04-13T03:33:51.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.145+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d41a23c0 con 0x7fa2d4105410 2024-04-13T03:33:51.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.145+0000 7fa2d1600640 1 -- 172.21.15.16:0/2068693004 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2bc0054e0 con 0x7fa2d4105410 2024-04-13T03:33:51.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.145+0000 7fa2d1600640 1 -- 172.21.15.16:0/2068693004 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc003bd0 con 0x7fa2d4105410 2024-04-13T03:33:51.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.145+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d41a2910 con 0x7fa2d4105410 2024-04-13T03:33:51.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.146+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2d410b540 con 0x7fa2d4105410 2024-04-13T03:33:51.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.149+0000 7fa2d1600640 1 -- 172.21.15.16:0/2068693004 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa2bc046020 con 0x7fa2d4105410 2024-04-13T03:33:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.151+0000 7fa2d1600640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa2b40a3bb0 0x7fa2b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.151+0000 7fa2d1600640 1 -- 172.21.15.16:0/2068693004 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa2bc0f2c90 con 0x7fa2d4105410 2024-04-13T03:33:51.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.151+0000 7fa2d3e00640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa2b40a3bb0 0x7fa2b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.152+0000 7fa2d3e00640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa2b40a3bb0 0x7fa2b40a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4105270 tx=0x7fa2c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:51.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.152+0000 7fa2d1600640 1 -- 172.21.15.16:0/2068693004 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2bc04d020 con 0x7fa2d4105410 2024-04-13T03:33:51.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.360+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa2d402d830 con 0x7fa2d4105410 2024-04-13T03:33:51.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.361+0000 7fa2d1600640 1 -- 172.21.15.16:0/2068693004 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fa2bc0bf240 con 0x7fa2d4105410 2024-04-13T03:33:51.363 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa2b40a3bb0 msgr2=0x7fa2b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa2b40a3bb0 0x7fa2b40a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4105270 tx=0x7fa2c4005f90 comp rx=0 tx=0).stop 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4105410 msgr2=0x7fa2d419aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4105410 0x7fa2d419aa50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc02c650 comp rx=0 tx=0).stop 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 shutdown_connections 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa2b40a3bb0 0x7fa2b40a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2d4105dd0 0x7fa2d41a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2d4105410 0x7fa2d419aa50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 --2- 172.21.15.16:0/2068693004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2d4104220 0x7fa2d419a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.364+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 >> 172.21.15.16:0/2068693004 conn(0x7fa2d40ff7b0 msgr2=0x7fa2d4108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.365+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 shutdown_connections 2024-04-13T03:33:51.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.365+0000 7fa2db200640 1 -- 172.21.15.16:0/2068693004 wait complete. 2024-04-13T03:33:51.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-13T03:33:51.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:51 smithi119 ceph-mon[35835]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-13T03:33:51.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:51 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3231386997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:33:51.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:51 smithi016 ceph-mon[31772]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-13T03:33:51.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:51 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3231386997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:33:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.614+0000 7fcfd6000640 1 -- 172.21.15.16:0/662753697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd010a1d0 msgr2=0x7fcfd0110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.614+0000 7fcfd6000640 1 --2- 172.21.15.16:0/662753697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd010a1d0 0x7fcfd0110a60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009ca0 tx=0x7fcfc802c100 comp rx=0 tx=0).stop 2024-04-13T03:33:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.615+0000 7fcfd6000640 1 -- 172.21.15.16:0/662753697 shutdown_connections 2024-04-13T03:33:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.615+0000 7fcfd6000640 1 --2- 172.21.15.16:0/662753697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd010a1d0 0x7fcfd0110a60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.615+0000 7fcfd6000640 1 --2- 172.21.15.16:0/662753697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfd0109810 0x7fcfd0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.615+0000 7fcfd6000640 1 --2- 172.21.15.16:0/662753697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd0108620 0x7fcfd0108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.615+0000 7fcfd6000640 1 -- 172.21.15.16:0/662753697 >> 172.21.15.16:0/662753697 conn(0x7fcfd00775c0 msgr2=0x7fcfd0079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.615+0000 7fcfd6000640 1 -- 172.21.15.16:0/662753697 shutdown_connections 2024-04-13T03:33:51.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.615+0000 7fcfd6000640 1 -- 172.21.15.16:0/662753697 wait complete. 2024-04-13T03:33:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.616+0000 7fcfd6000640 1 Processor -- start 2024-04-13T03:33:51.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.616+0000 7fcfd6000640 1 -- start start 2024-04-13T03:33:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfd6000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd0108620 0x7fcfd019e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfd6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd0109810 0x7fcfd019ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfd6000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfd010a1d0 0x7fcfd01a5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfd6000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01142b0 con 0x7fcfd0109810 2024-04-13T03:33:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfd6000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0114130 con 0x7fcfd0108620 2024-04-13T03:33:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfd6000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0114430 con 0x7fcfd010a1d0 2024-04-13T03:33:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfcea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd0109810 0x7fcfd019ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:51.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfcea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd0109810 0x7fcfd019ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59286/0 (socket says 172.21.15.16:59286) 2024-04-13T03:33:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfcea00640 1 -- 172.21.15.16:0/2580375997 learned_addr learned my addr 172.21.15.16:0/2580375997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfcf400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd0108620 0x7fcfd019e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:51.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfcfe00640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfd010a1d0 0x7fcfd01a5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfcea00640 1 -- 172.21.15.16:0/2580375997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfd010a1d0 msgr2=0x7fcfd01a5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfcea00640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfd010a1d0 0x7fcfd01a5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.617+0000 7fcfcea00640 1 -- 172.21.15.16:0/2580375997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd0108620 msgr2=0x7fcfd019e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.618+0000 7fcfcea00640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd0108620 0x7fcfd019e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.618+0000 7fcfcea00640 1 -- 172.21.15.16:0/2580375997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd01a6520 con 0x7fcfd0109810 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.618+0000 7fcfcf400640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd0108620 0x7fcfd019e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.618+0000 7fcfcea00640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd0109810 0x7fcfd019ee40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00e9f0 tx=0x7fcfbc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.618+0000 7fcfc7e00640 1 -- 172.21.15.16:0/2580375997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc00cd80 con 0x7fcfd0109810 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.618+0000 7fcfc7e00640 1 -- 172.21.15.16:0/2580375997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfbc005500 con 0x7fcfd0109810 2024-04-13T03:33:51.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.618+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd01a6810 con 0x7fcfd0109810 2024-04-13T03:33:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.618+0000 7fcfc7e00640 1 -- 172.21.15.16:0/2580375997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc010690 con 0x7fcfd0109810 2024-04-13T03:33:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.618+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd01a6d10 con 0x7fcfd0109810 2024-04-13T03:33:51.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.619+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd010f8e0 con 0x7fcfd0109810 2024-04-13T03:33:51.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:51 smithi188 ceph-mon[36398]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-13T03:33:51.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:51 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3231386997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:33:51.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.623+0000 7fcfc7e00640 1 -- 172.21.15.16:0/2580375997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fcfbc0046c0 con 0x7fcfd0109810 2024-04-13T03:33:51.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.625+0000 7fcfc7e00640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcfb40a3bb0 0x7fcfb40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:51.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.625+0000 7fcfc7e00640 1 -- 172.21.15.16:0/2580375997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcfbc0d0c30 con 0x7fcfd0109810 2024-04-13T03:33:51.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.625+0000 7fcfcf400640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcfb40a3bb0 0x7fcfb40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:51.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.626+0000 7fcfcf400640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcfb40a3bb0 0x7fcfb40a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0109670 tx=0x7fcfc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:51.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.630+0000 7fcfc7e00640 1 -- 172.21.15.16:0/2580375997 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcfbc09d1e0 con 0x7fcfd0109810 2024-04-13T03:33:51.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.840+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcfd0062bc0 con 0x7fcfd0109810 2024-04-13T03:33:51.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.841+0000 7fcfc7e00640 1 -- 172.21.15.16:0/2580375997 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fcfbc09cae0 con 0x7fcfd0109810 2024-04-13T03:33:51.843 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:33:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcfb40a3bb0 msgr2=0x7fcfb40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcfb40a3bb0 0x7fcfb40a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0109670 tx=0x7fcfc0005f90 comp rx=0 tx=0).stop 2024-04-13T03:33:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd0109810 msgr2=0x7fcfd019ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd0109810 0x7fcfd019ee40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00e9f0 tx=0x7fcfbc00edf0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 shutdown_connections 2024-04-13T03:33:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fcfb40a3bb0 0x7fcfb40a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcfd010a1d0 0x7fcfd01a5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd0109810 0x7fcfd019ee40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 --2- 172.21.15.16:0/2580375997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd0108620 0x7fcfd019e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.844+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 >> 172.21.15.16:0/2580375997 conn(0x7fcfd00775c0 msgr2=0x7fcfd010cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.845+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 shutdown_connections 2024-04-13T03:33:51.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:51.845+0000 7fcfd6000640 1 -- 172.21.15.16:0/2580375997 wait complete. 2024-04-13T03:33:51.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-13T03:33:52.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.096+0000 7fb624000640 1 -- 172.21.15.16:0/1492618991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c104220 msgr2=0x7fb61c104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.096+0000 7fb624000640 1 --2- 172.21.15.16:0/1492618991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c104220 0x7fb61c104640 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb610009900 tx=0x7fb61002bc30 comp rx=0 tx=0).stop 2024-04-13T03:33:52.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.096+0000 7fb624000640 1 -- 172.21.15.16:0/1492618991 shutdown_connections 2024-04-13T03:33:52.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.096+0000 7fb624000640 1 --2- 172.21.15.16:0/1492618991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb61c105dd0 0x7fb61c10c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.096+0000 7fb624000640 1 --2- 172.21.15.16:0/1492618991 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb61c105410 0x7fb61c105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.096+0000 7fb624000640 1 --2- 172.21.15.16:0/1492618991 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c104220 0x7fb61c104640 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.096+0000 7fb624000640 1 -- 172.21.15.16:0/1492618991 >> 172.21.15.16:0/1492618991 conn(0x7fb61c0ff7b0 msgr2=0x7fb61c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:52.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.096+0000 7fb624000640 1 -- 172.21.15.16:0/1492618991 shutdown_connections 2024-04-13T03:33:52.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.096+0000 7fb624000640 1 -- 172.21.15.16:0/1492618991 wait complete. 2024-04-13T03:33:52.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.097+0000 7fb624000640 1 Processor -- start 2024-04-13T03:33:52.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.097+0000 7fb624000640 1 -- start start 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.097+0000 7fb624000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb61c104220 0x7fb61c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.097+0000 7fb624000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c105410 0x7fb61c19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.097+0000 7fb624000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb61c105dd0 0x7fb61c1a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.097+0000 7fb624000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c10ff00 con 0x7fb61c105410 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.097+0000 7fb624000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c10fd80 con 0x7fb61c104220 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.097+0000 7fb624000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c110080 con 0x7fb61c105dd0 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb620c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c105410 0x7fb61c19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb621600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb61c104220 0x7fb61c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb622000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb61c105dd0 0x7fb61c1a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb621600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb61c104220 0x7fb61c19a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:35088/0 (socket says 172.21.15.16:35088) 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb620c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c105410 0x7fb61c19aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59314/0 (socket says 172.21.15.16:59314) 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb621600640 1 -- 172.21.15.16:0/638493826 learned_addr learned my addr 172.21.15.16:0/638493826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb621600640 1 -- 172.21.15.16:0/638493826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb61c105dd0 msgr2=0x7fb61c1a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb621600640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb61c105dd0 0x7fb61c1a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb621600640 1 -- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c105410 msgr2=0x7fb61c19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb621600640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c105410 0x7fb61c19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb621600640 1 -- 172.21.15.16:0/638493826 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb61c1a2150 con 0x7fb61c104220 2024-04-13T03:33:52.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb622000640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb61c105dd0 0x7fb61c1a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:52.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.098+0000 7fb620c00640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c105410 0x7fb61c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:52.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.099+0000 7fb621600640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb61c104220 0x7fb61c19a530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb610000c00 tx=0x7fb61002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.099+0000 7fb61a000640 1 -- 172.21.15.16:0/638493826 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb610043020 con 0x7fb61c104220 2024-04-13T03:33:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.099+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb61c1a23e0 con 0x7fb61c104220 2024-04-13T03:33:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.099+0000 7fb61a000640 1 -- 172.21.15.16:0/638493826 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6100054e0 con 0x7fb61c104220 2024-04-13T03:33:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.099+0000 7fb61a000640 1 -- 172.21.15.16:0/638493826 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb610003c40 con 0x7fb61c104220 2024-04-13T03:33:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.099+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb61c1a2980 con 0x7fb61c104220 2024-04-13T03:33:52.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.100+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb61c10b610 con 0x7fb61c104220 2024-04-13T03:33:52.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.103+0000 7fb61a000640 1 -- 172.21.15.16:0/638493826 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb610046020 con 0x7fb61c104220 2024-04-13T03:33:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.104+0000 7fb61a000640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6040a3c30 0x7fb6040a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.104+0000 7fb61a000640 1 -- 172.21.15.16:0/638493826 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb6100f2de0 con 0x7fb61c104220 2024-04-13T03:33:52.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.104+0000 7fb620c00640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6040a3c30 0x7fb6040a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:52.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.105+0000 7fb620c00640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6040a3c30 0x7fb6040a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb6080095f0 tx=0x7fb608005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:52.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.107+0000 7fb61a000640 1 -- 172.21.15.16:0/638493826 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb61004d020 con 0x7fb61c104220 2024-04-13T03:33:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.316+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb61c02d770 con 0x7fb61c104220 2024-04-13T03:33:52.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.317+0000 7fb61a000640 1 -- 172.21.15.16:0/638493826 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fb6100bf390 con 0x7fb61c104220 2024-04-13T03:33:52.319 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-13T03:33:52.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6040a3c30 msgr2=0x7fb6040a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6040a3c30 0x7fb6040a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb6080095f0 tx=0x7fb608005f90 comp rx=0 tx=0).stop 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb61c104220 msgr2=0x7fb61c19a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb61c104220 0x7fb61c19a530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb610000c00 tx=0x7fb61002c650 comp rx=0 tx=0).stop 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 shutdown_connections 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb6040a3c30 0x7fb6040a60f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb61c105dd0 0x7fb61c1a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb61c105410 0x7fb61c19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 --2- 172.21.15.16:0/638493826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb61c104220 0x7fb61c19a530 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.320+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 >> 172.21.15.16:0/638493826 conn(0x7fb61c0ff7b0 msgr2=0x7fb61c1087e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.321+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 shutdown_connections 2024-04-13T03:33:52.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.321+0000 7fb624000640 1 -- 172.21.15.16:0/638493826 wait complete. 2024-04-13T03:33:52.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-13T03:33:52.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.569+0000 7fe98aa00640 1 -- 172.21.15.16:0/2515568690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9841096f0 msgr2=0x7fe984109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.569+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2515568690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9841096f0 0x7fe984109b70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe970009900 tx=0x7fe97002bc30 comp rx=0 tx=0).stop 2024-04-13T03:33:52.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.570+0000 7fe98aa00640 1 -- 172.21.15.16:0/2515568690 shutdown_connections 2024-04-13T03:33:52.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.570+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2515568690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe98410a0b0 0x7fe984110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.570+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2515568690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9841096f0 0x7fe984109b70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.570+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2515568690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe984108500 0x7fe984108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.570+0000 7fe98aa00640 1 -- 172.21.15.16:0/2515568690 >> 172.21.15.16:0/2515568690 conn(0x7fe984077490 msgr2=0x7fe9840798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:52.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.570+0000 7fe98aa00640 1 -- 172.21.15.16:0/2515568690 shutdown_connections 2024-04-13T03:33:52.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.570+0000 7fe98aa00640 1 -- 172.21.15.16:0/2515568690 wait complete. 2024-04-13T03:33:52.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:52 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2068693004' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:33:52.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:52 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2580375997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:33:52.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.571+0000 7fe98aa00640 1 Processor -- start 2024-04-13T03:33:52.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.571+0000 7fe98aa00640 1 -- start start 2024-04-13T03:33:52.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.571+0000 7fe98aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe984108500 0x7fe98419e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:52.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.571+0000 7fe98aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9841096f0 0x7fe98419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:52.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.571+0000 7fe98aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe98410a0b0 0x7fe9841a5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:52.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.571+0000 7fe98aa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe984114190 con 0x7fe984108500 2024-04-13T03:33:52.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.571+0000 7fe98aa00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe984114010 con 0x7fe98410a0b0 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.571+0000 7fe98aa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe984114310 con 0x7fe9841096f0 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe988c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9841096f0 0x7fe98419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe989600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe984108500 0x7fe98419e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe988c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9841096f0 0x7fe98419ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53120/0 (socket says 172.21.15.16:53120) 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe988c00640 1 -- 172.21.15.16:0/2051135588 learned_addr learned my addr 172.21.15.16:0/2051135588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe98a000640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe98410a0b0 0x7fe9841a5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe98a000640 1 -- 172.21.15.16:0/2051135588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9841096f0 msgr2=0x7fe98419ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe98a000640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9841096f0 0x7fe98419ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe98a000640 1 -- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe984108500 msgr2=0x7fe98419e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe98a000640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe984108500 0x7fe98419e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe98a000640 1 -- 172.21.15.16:0/2051135588 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9841a63b0 con 0x7fe98410a0b0 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe988c00640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9841096f0 0x7fe98419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.572+0000 7fe989600640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe984108500 0x7fe98419e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.573+0000 7fe98a000640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe98410a0b0 0x7fe9841a5d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00d680 tx=0x7fe97c00db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.573+0000 7fe982000640 1 -- 172.21.15.16:0/2051135588 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c017070 con 0x7fe98410a0b0 2024-04-13T03:33:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.573+0000 7fe982000640 1 -- 172.21.15.16:0/2051135588 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe97c009820 con 0x7fe98410a0b0 2024-04-13T03:33:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.573+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9841a66a0 con 0x7fe98410a0b0 2024-04-13T03:33:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.573+0000 7fe982000640 1 -- 172.21.15.16:0/2051135588 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c004e00 con 0x7fe98410a0b0 2024-04-13T03:33:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.573+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9841a6c10 con 0x7fe98410a0b0 2024-04-13T03:33:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.574+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe98410f820 con 0x7fe98410a0b0 2024-04-13T03:33:52.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.576+0000 7fe982000640 1 -- 172.21.15.16:0/2051135588 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe97c0099d0 con 0x7fe98410a0b0 2024-04-13T03:33:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.578+0000 7fe982000640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe96c0a3c30 0x7fe96c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.578+0000 7fe982000640 1 -- 172.21.15.16:0/2051135588 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe97c0d0130 con 0x7fe98410a0b0 2024-04-13T03:33:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.578+0000 7fe989600640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe96c0a3c30 0x7fe96c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.579+0000 7fe989600640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe96c0a3c30 0x7fe96c0a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9780099a0 tx=0x7fe978009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.580+0000 7fe982000640 1 -- 172.21.15.16:0/2051135588 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe97c09c110 con 0x7fe98410a0b0 2024-04-13T03:33:52.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:52 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2068693004' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:33:52.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:52 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2580375997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:33:52.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:52 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2068693004' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:33:52.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:52 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2580375997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:33:52.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.795+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe984062bc0 con 0x7fe98410a0b0 2024-04-13T03:33:52.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.796+0000 7fe982000640 1 -- 172.21.15.16:0/2051135588 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fe97c0a0410 con 0x7fe98410a0b0 2024-04-13T03:33:52.797 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:33:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe96c0a3c30 msgr2=0x7fe96c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe96c0a3c30 0x7fe96c0a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9780099a0 tx=0x7fe978009500 comp rx=0 tx=0).stop 2024-04-13T03:33:52.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe98410a0b0 msgr2=0x7fe9841a5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe98410a0b0 0x7fe9841a5d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00d680 tx=0x7fe97c00db40 comp rx=0 tx=0).stop 2024-04-13T03:33:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 shutdown_connections 2024-04-13T03:33:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe96c0a3c30 0x7fe96c0a60f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe98410a0b0 0x7fe9841a5d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9841096f0 0x7fe98419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 --2- 172.21.15.16:0/2051135588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe984108500 0x7fe98419e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 >> 172.21.15.16:0/2051135588 conn(0x7fe984077490 msgr2=0x7fe98410c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 shutdown_connections 2024-04-13T03:33:52.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:52.799+0000 7fe98aa00640 1 -- 172.21.15.16:0/2051135588 wait complete. 2024-04-13T03:33:52.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:33:53.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.048+0000 7fe348000640 1 -- 172.21.15.16:0/727068922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe340105430 msgr2=0x7fe3401058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.048+0000 7fe348000640 1 --2- 172.21.15.16:0/727068922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe340105430 0x7fe3401058b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe330009860 tx=0x7fe33002bb70 comp rx=0 tx=0).stop 2024-04-13T03:33:53.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.049+0000 7fe348000640 1 -- 172.21.15.16:0/727068922 shutdown_connections 2024-04-13T03:33:53.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.049+0000 7fe348000640 1 --2- 172.21.15.16:0/727068922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe340105df0 0x7fe34010c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.049+0000 7fe348000640 1 --2- 172.21.15.16:0/727068922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe340105430 0x7fe3401058b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.049+0000 7fe348000640 1 --2- 172.21.15.16:0/727068922 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe340104240 0x7fe340104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.049+0000 7fe348000640 1 -- 172.21.15.16:0/727068922 >> 172.21.15.16:0/727068922 conn(0x7fe3400ff7b0 msgr2=0x7fe340101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:53.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.049+0000 7fe348000640 1 -- 172.21.15.16:0/727068922 shutdown_connections 2024-04-13T03:33:53.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.049+0000 7fe348000640 1 -- 172.21.15.16:0/727068922 wait complete. 2024-04-13T03:33:53.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.050+0000 7fe348000640 1 Processor -- start 2024-04-13T03:33:53.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.050+0000 7fe348000640 1 -- start start 2024-04-13T03:33:53.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe348000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe340104240 0x7fe34019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe348000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe340105430 0x7fe34019aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe348000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe340105df0 0x7fe3401a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe348000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe34010fed0 con 0x7fe340105df0 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe348000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe34010fd50 con 0x7fe340105430 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe348000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe340110050 con 0x7fe340104240 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe345600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe340104240 0x7fe34019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe345600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe340104240 0x7fe34019a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53140/0 (socket says 172.21.15.16:53140) 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe344c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe340105430 0x7fe34019aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe345600640 1 -- 172.21.15.16:0/1190983348 learned_addr learned my addr 172.21.15.16:0/1190983348 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.051+0000 7fe346000640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe340105df0 0x7fe3401a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe345600640 1 -- 172.21.15.16:0/1190983348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe340105430 msgr2=0x7fe34019aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe345600640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe340105430 0x7fe34019aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe345600640 1 -- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe340105df0 msgr2=0x7fe3401a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe345600640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe340105df0 0x7fe3401a1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe345600640 1 -- 172.21.15.16:0/1190983348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3401a2120 con 0x7fe340104240 2024-04-13T03:33:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe344c00640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe340105430 0x7fe34019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe346000640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe340105df0 0x7fe3401a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:33:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe345600640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe340104240 0x7fe34019a500 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe33400e9f0 tx=0x7fe33400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:53.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe33e000640 1 -- 172.21.15.16:0/1190983348 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33400ccc0 con 0x7fe340104240 2024-04-13T03:33:53.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3401a2410 con 0x7fe340104240 2024-04-13T03:33:53.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe33e000640 1 -- 172.21.15.16:0/1190983348 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3340035f0 con 0x7fe340104240 2024-04-13T03:33:53.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.052+0000 7fe33e000640 1 -- 172.21.15.16:0/1190983348 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33400ce60 con 0x7fe340104240 2024-04-13T03:33:53.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.053+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3401a2980 con 0x7fe340104240 2024-04-13T03:33:53.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.054+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe34010b5e0 con 0x7fe340104240 2024-04-13T03:33:53.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.056+0000 7fe33e000640 1 -- 172.21.15.16:0/1190983348 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe3340037a0 con 0x7fe340104240 2024-04-13T03:33:53.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.057+0000 7fe33e000640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3280a3bb0 0x7fe3280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:53.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.058+0000 7fe33e000640 1 -- 172.21.15.16:0/1190983348 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe3340d0b10 con 0x7fe340104240 2024-04-13T03:33:53.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.058+0000 7fe344c00640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3280a3bb0 0x7fe3280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:53.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.059+0000 7fe344c00640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3280a3bb0 0x7fe3280a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe33002c180 tx=0x7fe33002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:53.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.062+0000 7fe33e000640 1 -- 172.21.15.16:0/1190983348 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe33409d0c0 con 0x7fe340104240 2024-04-13T03:33:53.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.315+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe34002d5e0 con 0x7fe340104240 2024-04-13T03:33:53.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.317+0000 7fe33e000640 1 -- 172.21.15.16:0/1190983348 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11934 (secure 0 0 0) 0x7fe33409c9c0 con 0x7fe340104240 2024-04-13T03:33:53.318 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:53.318 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:32:07.796132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24530,"mds_2":24523,"mds_3":24505,"mds_4":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24505,24500,24587,24523,24530]},"id":1}]} 2024-04-13T03:33:53.319 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 26 2024-04-13T03:33:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3280a3bb0 msgr2=0x7fe3280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3280a3bb0 0x7fe3280a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe33002c180 tx=0x7fe33002e040 comp rx=0 tx=0).stop 2024-04-13T03:33:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe340104240 msgr2=0x7fe34019a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe340104240 0x7fe34019a500 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe33400e9f0 tx=0x7fe33400eeb0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 shutdown_connections 2024-04-13T03:33:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe3280a3bb0 0x7fe3280a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe340105df0 0x7fe3401a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe340105430 0x7fe34019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 --2- 172.21.15.16:0/1190983348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe340104240 0x7fe34019a500 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 >> 172.21.15.16:0/1190983348 conn(0x7fe3400ff7b0 msgr2=0x7fe3401087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 shutdown_connections 2024-04-13T03:33:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.320+0000 7fe348000640 1 -- 172.21.15.16:0/1190983348 wait complete. 2024-04-13T03:33:53.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:33:53.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.556+0000 7f0db8a00640 1 -- 172.21.15.16:0/2486691561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 msgr2=0x7f0db4074e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.556+0000 7f0db8a00640 1 --2- 172.21.15.16:0/2486691561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 0x7f0db4074e00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0da0009900 tx=0x7f0da002bcd0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.557+0000 7f0db8a00640 1 -- 172.21.15.16:0/2486691561 shutdown_connections 2024-04-13T03:33:53.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.557+0000 7f0db8a00640 1 --2- 172.21.15.16:0/2486691561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db4109d10 0x7f0db410c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.557+0000 7f0db8a00640 1 --2- 172.21.15.16:0/2486691561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 0x7f0db4074e00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.557+0000 7f0db8a00640 1 --2- 172.21.15.16:0/2486691561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0db4075f80 0x7f0db40743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.557+0000 7f0db8a00640 1 -- 172.21.15.16:0/2486691561 >> 172.21.15.16:0/2486691561 conn(0x7f0db40fd3d0 msgr2=0x7f0db40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.557+0000 7f0db8a00640 1 -- 172.21.15.16:0/2486691561 shutdown_connections 2024-04-13T03:33:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.557+0000 7f0db8a00640 1 -- 172.21.15.16:0/2486691561 wait complete. 2024-04-13T03:33:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.558+0000 7f0db8a00640 1 Processor -- start 2024-04-13T03:33:53.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.558+0000 7f0db8a00640 1 -- start start 2024-04-13T03:33:53.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 0x7f0db419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db4075f80 0x7f0db419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db8a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0db4109d10 0x7f0db41a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db8a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0db410fd00 con 0x7f0db40749a0 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db8a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0db410fb80 con 0x7f0db4109d10 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db8a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0db410fe80 con 0x7f0db4075f80 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 0x7f0db419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 0x7f0db419a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59356/0 (socket says 172.21.15.16:59356) 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db3400640 1 -- 172.21.15.16:0/1317600712 learned_addr learned my addr 172.21.15.16:0/1317600712 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.559+0000 7f0db2a00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db4075f80 0x7f0db419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0db3e00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0db4109d10 0x7f0db41a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0db3400640 1 -- 172.21.15.16:0/1317600712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db4075f80 msgr2=0x7f0db419a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0db3400640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db4075f80 0x7f0db419a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0db3400640 1 -- 172.21.15.16:0/1317600712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0db4109d10 msgr2=0x7f0db41a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0db3400640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0db4109d10 0x7f0db41a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0db3400640 1 -- 172.21.15.16:0/1317600712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db41a1ef0 con 0x7f0db40749a0 2024-04-13T03:33:53.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0db3400640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 0x7f0db419a2d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0da400e9f0 tx=0x7f0da400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:53.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0dabe00640 1 -- 172.21.15.16:0/1317600712 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da400cd80 con 0x7f0db40749a0 2024-04-13T03:33:53.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0dabe00640 1 -- 172.21.15.16:0/1317600712 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da4010910 con 0x7f0db40749a0 2024-04-13T03:33:53.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db41a21e0 con 0x7f0db40749a0 2024-04-13T03:33:53.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.560+0000 7f0dabe00640 1 -- 172.21.15.16:0/1317600712 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da4018990 con 0x7f0db40749a0 2024-04-13T03:33:53.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.561+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db41a2700 con 0x7f0db40749a0 2024-04-13T03:33:53.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.562+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0db4108ad0 con 0x7f0db40749a0 2024-04-13T03:33:53.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.564+0000 7f0dabe00640 1 -- 172.21.15.16:0/1317600712 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0da4010ac0 con 0x7f0db40749a0 2024-04-13T03:33:53.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.565+0000 7f0dabe00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0d980a3bb0 0x7f0d980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:53.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.565+0000 7f0dabe00640 1 -- 172.21.15.16:0/1317600712 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0da40d0c00 con 0x7f0db40749a0 2024-04-13T03:33:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.566+0000 7f0db2a00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0d980a3bb0 0x7f0d980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.566+0000 7f0db2a00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0d980a3bb0 0x7f0d980a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0da002c850 tx=0x7f0da0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:53.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:53 smithi119 ceph-mon[35835]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-13T03:33:53.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:53 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/638493826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:33:53.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:53 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2051135588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:33:53.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.571+0000 7f0dabe00640 1 -- 172.21.15.16:0/1317600712 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0da409d1b0 con 0x7f0db40749a0 2024-04-13T03:33:53.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:53 smithi016 ceph-mon[31772]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-13T03:33:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:53 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/638493826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:33:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:53 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2051135588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:33:53.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:53 smithi188 ceph-mon[36398]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-13T03:33:53.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:53 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/638493826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:33:53.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:53 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2051135588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:33:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.783+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0db4062bc0 con 0x7f0db40749a0 2024-04-13T03:33:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.786+0000 7f0dabe00640 1 -- 172.21.15.16:0/1317600712 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24874 (secure 0 0 0) 0x7f0da409cab0 con 0x7f0db40749a0 2024-04-13T03:33:53.788 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:53.788 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":94,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:33:42.153326+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:33:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.789+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0d980a3bb0 msgr2=0x7f0d980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.789+0000 7f0db8a00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0d980a3bb0 0x7f0d980a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0da002c850 tx=0x7f0da0000bc0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.789+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 msgr2=0x7f0db419a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.789+0000 7f0db8a00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 0x7f0db419a2d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0da400e9f0 tx=0x7f0da400eeb0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.790+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 shutdown_connections 2024-04-13T03:33:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.790+0000 7f0db8a00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0d980a3bb0 0x7f0d980a6070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.790+0000 7f0db8a00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0db4109d10 0x7f0db41a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.790+0000 7f0db8a00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0db4075f80 0x7f0db419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.790+0000 7f0db8a00640 1 --2- 172.21.15.16:0/1317600712 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db40749a0 0x7f0db419a2d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:53.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.790+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 >> 172.21.15.16:0/1317600712 conn(0x7f0db40fd3d0 msgr2=0x7f0db4105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:53.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.790+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 shutdown_connections 2024-04-13T03:33:53.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:53.790+0000 7f0db8a00640 1 -- 172.21.15.16:0/1317600712 wait complete. 2024-04-13T03:33:53.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-13T03:31:34.802688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-13T03:31:36.036786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-13T03:31:39.824413+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-13T03:33:53.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:33:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.012+0000 7f6778a00640 1 -- 172.21.15.16:0/1672223001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6774101880 msgr2=0x7f6774101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.012+0000 7f6778a00640 1 --2- 172.21.15.16:0/1672223001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6774101880 0x7f6774101d00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f675c009930 tx=0x7f675c02bc20 comp rx=0 tx=0).stop 2024-04-13T03:33:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.013+0000 7f6778a00640 1 -- 172.21.15.16:0/1672223001 shutdown_connections 2024-04-13T03:33:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.013+0000 7f6778a00640 1 --2- 172.21.15.16:0/1672223001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6774106580 0x7f6774108970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.013+0000 7f6778a00640 1 --2- 172.21.15.16:0/1672223001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6774101880 0x7f6774101d00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.013+0000 7f6778a00640 1 --2- 172.21.15.16:0/1672223001 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774068470 0x7f6774101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.013+0000 7f6778a00640 1 -- 172.21.15.16:0/1672223001 >> 172.21.15.16:0/1672223001 conn(0x7f67740fb6c0 msgr2=0x7f67740fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:54.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.013+0000 7f6778a00640 1 -- 172.21.15.16:0/1672223001 shutdown_connections 2024-04-13T03:33:54.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.013+0000 7f6778a00640 1 -- 172.21.15.16:0/1672223001 wait complete. 2024-04-13T03:33:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.014+0000 7f6778a00640 1 Processor -- start 2024-04-13T03:33:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.014+0000 7f6778a00640 1 -- start start 2024-04-13T03:33:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6778a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6774068470 0x7f67740fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6778a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774101880 0x7f67740fbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6778a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6774106580 0x7f67740fc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6778a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f677410ba00 con 0x7f6774101880 2024-04-13T03:33:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6778a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f677410b880 con 0x7f6774106580 2024-04-13T03:33:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6778a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f677410bb80 con 0x7f6774068470 2024-04-13T03:33:54.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6769e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774101880 0x7f67740fbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6769e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774101880 0x7f67740fbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59372/0 (socket says 172.21.15.16:59372) 2024-04-13T03:33:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6769e00640 1 -- 172.21.15.16:0/4168995331 learned_addr learned my addr 172.21.15.16:0/4168995331 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6773200640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6774106580 0x7f67740fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.015+0000 7f6772800640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6774068470 0x7f67740fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f6773200640 1 -- 172.21.15.16:0/4168995331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6774068470 msgr2=0x7f67740fd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f6773200640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6774068470 0x7f67740fd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f6773200640 1 -- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774101880 msgr2=0x7f67740fbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f6773200640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774101880 0x7f67740fbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f6773200640 1 -- 172.21.15.16:0/4168995331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67740fcd00 con 0x7f6774106580 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f6769e00640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774101880 0x7f67740fbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f6773200640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6774106580 0x7f67740fc470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f676c009fd0 tx=0x7f676c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f676be00640 1 -- 172.21.15.16:0/4168995331 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c00ce20 con 0x7f6774106580 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f676be00640 1 -- 172.21.15.16:0/4168995331 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f676c005020 con 0x7f6774106580 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67741a2360 con 0x7f6774106580 2024-04-13T03:33:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f676be00640 1 -- 172.21.15.16:0/4168995331 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c010720 con 0x7f6774106580 2024-04-13T03:33:54.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.016+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67741a2880 con 0x7f6774106580 2024-04-13T03:33:54.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.017+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6774069150 con 0x7f6774106580 2024-04-13T03:33:54.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.020+0000 7f676be00640 1 -- 172.21.15.16:0/4168995331 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f676c005540 con 0x7f6774106580 2024-04-13T03:33:54.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.021+0000 7f676be00640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f67580a3bb0 0x7f67580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.021+0000 7f676be00640 1 -- 172.21.15.16:0/4168995331 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f676c0d0f50 con 0x7f6774106580 2024-04-13T03:33:54.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.022+0000 7f6772800640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f67580a3bb0 0x7f67580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.022+0000 7f6772800640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f67580a3bb0 0x7f67580a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f67741011a0 tx=0x7f6764010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:54.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.024+0000 7f676be00640 1 -- 172.21.15.16:0/4168995331 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f676c09d590 con 0x7f6774106580 2024-04-13T03:33:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.235+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67741a2d60 con 0x7f6774106580 2024-04-13T03:33:54.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.236+0000 7f676be00640 1 -- 172.21.15.16:0/4168995331 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f676c09ce90 con 0x7f6774106580 2024-04-13T03:33:54.237 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:33:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f67580a3bb0 msgr2=0x7f67580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f67580a3bb0 0x7f67580a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f67741011a0 tx=0x7f6764010040 comp rx=0 tx=0).stop 2024-04-13T03:33:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6774106580 msgr2=0x7f67740fc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6774106580 0x7f67740fc470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f676c009fd0 tx=0x7f676c00ef10 comp rx=0 tx=0).stop 2024-04-13T03:33:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 shutdown_connections 2024-04-13T03:33:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f67580a3bb0 0x7f67580a6070 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6774106580 0x7f67740fc470 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774101880 0x7f67740fbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 --2- 172.21.15.16:0/4168995331 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6774068470 0x7f67740fd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 >> 172.21.15.16:0/4168995331 conn(0x7f67740fb6c0 msgr2=0x7f6774108500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 shutdown_connections 2024-04-13T03:33:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.239+0000 7f6778a00640 1 -- 172.21.15.16:0/4168995331 wait complete. 2024-04-13T03:33:54.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-13T03:33:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.489+0000 7f9707e00640 1 -- 172.21.15.16:0/4039751303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97081052f0 msgr2=0x7f9708105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.489+0000 7f9707e00640 1 --2- 172.21.15.16:0/4039751303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97081052f0 0x7f9708105770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f96f0009860 tx=0x7f96f002bb70 comp rx=0 tx=0).stop 2024-04-13T03:33:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.489+0000 7f9707e00640 1 -- 172.21.15.16:0/4039751303 shutdown_connections 2024-04-13T03:33:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.489+0000 7f9707e00640 1 --2- 172.21.15.16:0/4039751303 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9708105cb0 0x7f970810c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.489+0000 7f9707e00640 1 --2- 172.21.15.16:0/4039751303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97081052f0 0x7f9708105770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.489+0000 7f9707e00640 1 --2- 172.21.15.16:0/4039751303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9708104100 0x7f9708104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.489+0000 7f9707e00640 1 -- 172.21.15.16:0/4039751303 >> 172.21.15.16:0/4039751303 conn(0x7f97080ff670 msgr2=0x7f9708101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.489+0000 7f9707e00640 1 -- 172.21.15.16:0/4039751303 shutdown_connections 2024-04-13T03:33:54.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.489+0000 7f9707e00640 1 -- 172.21.15.16:0/4039751303 wait complete. 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.490+0000 7f9707e00640 1 Processor -- start 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707e00640 1 -- start start 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9708104100 0x7f970819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97081052f0 0x7f970819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708105cb0 0x7f97081a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f970810fd90 con 0x7f9708105cb0 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f970810fc10 con 0x7f9708104100 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f970810ff10 con 0x7f97081052f0 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708105cb0 0x7f97081a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708105cb0 0x7f97081a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59388/0 (socket says 172.21.15.16:59388) 2024-04-13T03:33:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9707400640 1 -- 172.21.15.16:0/1292533997 learned_addr learned my addr 172.21.15.16:0/1292533997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f9706a00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9708104100 0x7f970819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.491+0000 7f96ffe00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97081052f0 0x7f970819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f9707400640 1 -- 172.21.15.16:0/1292533997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97081052f0 msgr2=0x7f970819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f9707400640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97081052f0 0x7f970819a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f9707400640 1 -- 172.21.15.16:0/1292533997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9708104100 msgr2=0x7f970819a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f9707400640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9708104100 0x7f970819a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f9707400640 1 -- 172.21.15.16:0/1292533997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97081a1fe0 con 0x7f9708105cb0 2024-04-13T03:33:54.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f9707400640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708105cb0 0x7f97081a1970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f970000eb10 tx=0x7f970000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:54.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f96ff400640 1 -- 172.21.15.16:0/1292533997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700004b20 con 0x7f9708105cb0 2024-04-13T03:33:54.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f96ff400640 1 -- 172.21.15.16:0/1292533997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9700004cc0 con 0x7f9708105cb0 2024-04-13T03:33:54.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f96ff400640 1 -- 172.21.15.16:0/1292533997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700010760 con 0x7f9708105cb0 2024-04-13T03:33:54.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.492+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97081a22d0 con 0x7f9708105cb0 2024-04-13T03:33:54.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.493+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97081a27a0 con 0x7f9708105cb0 2024-04-13T03:33:54.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.494+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f970810b420 con 0x7f9708105cb0 2024-04-13T03:33:54.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.496+0000 7f96ff400640 1 -- 172.21.15.16:0/1292533997 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9700005000 con 0x7f9708105cb0 2024-04-13T03:33:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.497+0000 7f96ff400640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.497+0000 7f96ff400640 1 -- 172.21.15.16:0/1292533997 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f97000d11e0 con 0x7f9708105cb0 2024-04-13T03:33:54.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.498+0000 7f9706a00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.498+0000 7f9706a00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009530 tx=0x7f96f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:54.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.500+0000 7f96ff400640 1 -- 172.21.15.16:0/1292533997 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f970009d790 con 0x7f9708105cb0 2024-04-13T03:33:54.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:54 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1190983348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:33:54.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:54 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1317600712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:33:54.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:54 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4168995331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:33:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:54 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1190983348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:33:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:54 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1317600712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:33:54.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:54 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4168995331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:33:54.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:54 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1190983348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:33:54.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:54 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1317600712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:33:54.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:54 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4168995331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:33:54.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.710+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f970802d890 con 0x7f9708105cb0 2024-04-13T03:33:54.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.711+0000 7f96ff400640 1 -- 172.21.15.16:0/1292533997 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f970009d090 con 0x7f9708105cb0 2024-04-13T03:33:54.713 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:33:54.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f96ec0a3c30 msgr2=0x7f96ec0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009530 tx=0x7f96f8009260 comp rx=0 tx=0).stop 2024-04-13T03:33:54.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708105cb0 msgr2=0x7f97081a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708105cb0 0x7f97081a1970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f970000eb10 tx=0x7f970000c2d0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 shutdown_connections 2024-04-13T03:33:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9708105cb0 0x7f97081a1970 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97081052f0 0x7f970819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 --2- 172.21.15.16:0/1292533997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9708104100 0x7f970819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.714+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 >> 172.21.15.16:0/1292533997 conn(0x7f97080ff670 msgr2=0x7f97081085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.715+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 shutdown_connections 2024-04-13T03:33:54.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.715+0000 7f9707e00640 1 -- 172.21.15.16:0/1292533997 wait complete. 2024-04-13T03:33:54.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-13T03:33:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.961+0000 7f1fe3200640 1 -- 172.21.15.16:0/801504912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103080 msgr2=0x7f1fe4103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.961+0000 7f1fe3200640 1 --2- 172.21.15.16:0/801504912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103080 0x7f1fe4103500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009900 tx=0x7f1fd002bc30 comp rx=0 tx=0).stop 2024-04-13T03:33:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.961+0000 7f1fe3200640 1 -- 172.21.15.16:0/801504912 shutdown_connections 2024-04-13T03:33:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.961+0000 7f1fe3200640 1 --2- 172.21.15.16:0/801504912 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fe4103a40 0x7f1fe410a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.961+0000 7f1fe3200640 1 --2- 172.21.15.16:0/801504912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103080 0x7f1fe4103500 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.961+0000 7f1fe3200640 1 --2- 172.21.15.16:0/801504912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe4101ea0 0x7f1fe41022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.961+0000 7f1fe3200640 1 -- 172.21.15.16:0/801504912 >> 172.21.15.16:0/801504912 conn(0x7f1fe40fd460 msgr2=0x7f1fe40ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.961+0000 7f1fe3200640 1 -- 172.21.15.16:0/801504912 shutdown_connections 2024-04-13T03:33:54.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.961+0000 7f1fe3200640 1 -- 172.21.15.16:0/801504912 wait complete. 2024-04-13T03:33:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.963+0000 7f1fe3200640 1 Processor -- start 2024-04-13T03:33:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.963+0000 7f1fe3200640 1 -- start start 2024-04-13T03:33:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.963+0000 7f1fe3200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fe4101ea0 0x7f1fe4075d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.963+0000 7f1fe3200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe4103080 0x7f1fe40743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.963+0000 7f1fe3200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103a40 0x7f1fe40748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.963+0000 7f1fe3200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe410dbf0 con 0x7f1fe4103a40 2024-04-13T03:33:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.963+0000 7f1fe3200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe410da70 con 0x7f1fe4103080 2024-04-13T03:33:54.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.963+0000 7f1fe3200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe410dd70 con 0x7f1fe4101ea0 2024-04-13T03:33:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103a40 0x7f1fe40748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103a40 0x7f1fe40748f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59404/0 (socket says 172.21.15.16:59404) 2024-04-13T03:33:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe2800640 1 -- 172.21.15.16:0/3113405836 learned_addr learned my addr 172.21.15.16:0/3113405836 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe1400640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe4103080 0x7f1fe40743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe1e00640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fe4101ea0 0x7f1fe4075d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe2800640 1 -- 172.21.15.16:0/3113405836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fe4101ea0 msgr2=0x7f1fe4075d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe2800640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fe4101ea0 0x7f1fe4075d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe2800640 1 -- 172.21.15.16:0/3113405836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe4103080 msgr2=0x7f1fe40743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe2800640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe4103080 0x7f1fe40743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe2800640 1 -- 172.21.15.16:0/3113405836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe4075180 con 0x7f1fe4103a40 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.964+0000 7f1fe1e00640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fe4101ea0 0x7f1fe4075d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.965+0000 7f1fe2800640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103a40 0x7f1fe40748f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00eb10 tx=0x7f1fdc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.965+0000 7f1fdaa00640 1 -- 172.21.15.16:0/3113405836 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc004b20 con 0x7f1fe4103a40 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.965+0000 7f1fdaa00640 1 -- 172.21.15.16:0/3113405836 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fdc004cc0 con 0x7f1fe4103a40 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.965+0000 7f1fdaa00640 1 -- 172.21.15.16:0/3113405836 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc010760 con 0x7f1fe4103a40 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.965+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe41a22d0 con 0x7f1fe4103a40 2024-04-13T03:33:54.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.965+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe41a2840 con 0x7f1fe4103a40 2024-04-13T03:33:54.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.966+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe41091e0 con 0x7f1fe4103a40 2024-04-13T03:33:54.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.969+0000 7f1fdaa00640 1 -- 172.21.15.16:0/3113405836 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1fdc005020 con 0x7f1fe4103a40 2024-04-13T03:33:54.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.971+0000 7f1fdaa00640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1fc80a3c30 0x7f1fc80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.972+0000 7f1fdaa00640 1 -- 172.21.15.16:0/3113405836 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1fdc0d0ed0 con 0x7f1fe4103a40 2024-04-13T03:33:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.972+0000 7f1fe1e00640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1fc80a3c30 0x7f1fc80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:54.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.972+0000 7f1fe1e00640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1fc80a3c30 0x7f1fc80a60f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4005d00 tx=0x7f1fd4005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:54.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:54.976+0000 7f1fdaa00640 1 -- 172.21.15.16:0/3113405836 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1fdc09d480 con 0x7f1fe4103a40 2024-04-13T03:33:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.185+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1fe4062bc0 con 0x7f1fe4103a40 2024-04-13T03:33:55.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.186+0000 7f1fdaa00640 1 -- 172.21.15.16:0/3113405836 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f1fdc09cd80 con 0x7f1fe4103a40 2024-04-13T03:33:55.188 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-13T03:33:55.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1fc80a3c30 msgr2=0x7f1fc80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1fc80a3c30 0x7f1fc80a60f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4005d00 tx=0x7f1fd4005c90 comp rx=0 tx=0).stop 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103a40 msgr2=0x7f1fe40748f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103a40 0x7f1fe40748f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00eb10 tx=0x7f1fdc00c2d0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 shutdown_connections 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1fc80a3c30 0x7f1fc80a60f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4103a40 0x7f1fe40748f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe4103080 0x7f1fe40743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 --2- 172.21.15.16:0/3113405836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1fe4101ea0 0x7f1fe4075d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 >> 172.21.15.16:0/3113405836 conn(0x7f1fe40fd460 msgr2=0x7f1fe41063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.189+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 shutdown_connections 2024-04-13T03:33:55.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.190+0000 7f1fe3200640 1 -- 172.21.15.16:0/3113405836 wait complete. 2024-04-13T03:33:55.209 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-13T03:33:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.425+0000 7f3b34c00640 1 -- 172.21.15.16:0/998204977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 msgr2=0x7f3b30110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.425+0000 7f3b34c00640 1 --2- 172.21.15.16:0/998204977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 0x7f3b30110a60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009ca0 tx=0x7f3b2802c0f0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.425+0000 7f3b34c00640 1 -- 172.21.15.16:0/998204977 shutdown_connections 2024-04-13T03:33:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.425+0000 7f3b34c00640 1 --2- 172.21.15.16:0/998204977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 0x7f3b30110a60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.425+0000 7f3b34c00640 1 --2- 172.21.15.16:0/998204977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b30109810 0x7f3b30109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.425+0000 7f3b34c00640 1 --2- 172.21.15.16:0/998204977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b30108620 0x7f3b30108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.425+0000 7f3b34c00640 1 -- 172.21.15.16:0/998204977 >> 172.21.15.16:0/998204977 conn(0x7f3b300775c0 msgr2=0x7f3b30079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.425+0000 7f3b34c00640 1 -- 172.21.15.16:0/998204977 shutdown_connections 2024-04-13T03:33:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.425+0000 7f3b34c00640 1 -- 172.21.15.16:0/998204977 wait complete. 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.426+0000 7f3b34c00640 1 Processor -- start 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.426+0000 7f3b34c00640 1 -- start start 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b34c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b30108620 0x7f3b3019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b34c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b30109810 0x7f3b3019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b34c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 0x7f3b301a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b34c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3b301142b0 con 0x7f3b3010a1d0 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b34c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30114130 con 0x7f3b30108620 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b34c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30114430 con 0x7f3b30109810 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b2e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 0x7f3b301a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:55.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b2d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b30109810 0x7f3b3019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b2d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b30109810 0x7f3b3019ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53216/0 (socket says 172.21.15.16:53216) 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b2e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 0x7f3b301a5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59426/0 (socket says 172.21.15.16:59426) 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b2d400640 1 -- 172.21.15.16:0/2138796587 learned_addr learned my addr 172.21.15.16:0/2138796587 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b2de00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b30108620 0x7f3b3019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.427+0000 7f3b2d400640 1 -- 172.21.15.16:0/2138796587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b30108620 msgr2=0x7f3b3019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b2d400640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b30108620 0x7f3b3019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b2d400640 1 -- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 msgr2=0x7f3b301a5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b2d400640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 0x7f3b301a5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b2d400640 1 -- 172.21.15.16:0/2138796587 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b301a6540 con 0x7f3b30109810 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b2e800640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 0x7f3b301a5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:55.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b2d400640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b30109810 0x7f3b3019ee60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00e9f0 tx=0x7f3b1c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b26a00640 1 -- 172.21.15.16:0/2138796587 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c00cd80 con 0x7f3b30109810 2024-04-13T03:33:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b26a00640 1 -- 172.21.15.16:0/2138796587 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b1c005500 con 0x7f3b30109810 2024-04-13T03:33:55.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b301a6830 con 0x7f3b30109810 2024-04-13T03:33:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.428+0000 7f3b26a00640 1 -- 172.21.15.16:0/2138796587 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c010690 con 0x7f3b30109810 2024-04-13T03:33:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.429+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b301a6d30 con 0x7f3b30109810 2024-04-13T03:33:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.430+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b3010f940 con 0x7f3b30109810 2024-04-13T03:33:55.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.431+0000 7f3b26a00640 1 -- 172.21.15.16:0/2138796587 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3b1c0046c0 con 0x7f3b30109810 2024-04-13T03:33:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.433+0000 7f3b26a00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3b140a3bb0 0x7f3b140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:55.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.434+0000 7f3b26a00640 1 -- 172.21.15.16:0/2138796587 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3b1c0d0be0 con 0x7f3b30109810 2024-04-13T03:33:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.434+0000 7f3b2de00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3b140a3bb0 0x7f3b140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:55.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.434+0000 7f3b2de00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3b140a3bb0 0x7f3b140a6070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3b200095f0 tx=0x7f3b20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:55.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.435+0000 7f3b26a00640 1 -- 172.21.15.16:0/2138796587 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b1c09d190 con 0x7f3b30109810 2024-04-13T03:33:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:55 smithi119 ceph-mon[35835]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:33:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:55 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1292533997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:33:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:55 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3113405836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:33:55.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:55 smithi016 ceph-mon[31772]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:33:55.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:55 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1292533997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:33:55.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:55 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3113405836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:33:55.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:55 smithi188 ceph-mon[36398]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:33:55.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:55 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1292533997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:33:55.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:55 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3113405836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:33:55.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.633+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3b3002d770 con 0x7f3b30109810 2024-04-13T03:33:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.634+0000 7f3b26a00640 1 -- 172.21.15.16:0/2138796587 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f3b1c0bd0d0 con 0x7f3b30109810 2024-04-13T03:33:55.635 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:33:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3b140a3bb0 msgr2=0x7f3b140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3b140a3bb0 0x7f3b140a6070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3b200095f0 tx=0x7f3b20005f90 comp rx=0 tx=0).stop 2024-04-13T03:33:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b30109810 msgr2=0x7f3b3019ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b30109810 0x7f3b3019ee60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00e9f0 tx=0x7f3b1c00edf0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 shutdown_connections 2024-04-13T03:33:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f3b140a3bb0 0x7f3b140a6070 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3010a1d0 0x7f3b301a5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b30109810 0x7f3b3019ee60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 --2- 172.21.15.16:0/2138796587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b30108620 0x7f3b3019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 >> 172.21.15.16:0/2138796587 conn(0x7f3b300775c0 msgr2=0x7f3b3010cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 shutdown_connections 2024-04-13T03:33:55.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.637+0000 7f3b34c00640 1 -- 172.21.15.16:0/2138796587 wait complete. 2024-04-13T03:33:55.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:33:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.872+0000 7f05e6e00640 1 -- 172.21.15.16:0/1088798720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05e0105230 msgr2=0x7f05e01056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.872+0000 7f05e6e00640 1 --2- 172.21.15.16:0/1088798720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05e0105230 0x7f05e01056b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f05d4009ca0 tx=0x7f05d402c100 comp rx=0 tx=0).stop 2024-04-13T03:33:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.872+0000 7f05e6e00640 1 -- 172.21.15.16:0/1088798720 shutdown_connections 2024-04-13T03:33:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.872+0000 7f05e6e00640 1 --2- 172.21.15.16:0/1088798720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05e0105bf0 0x7f05e010c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.872+0000 7f05e6e00640 1 --2- 172.21.15.16:0/1088798720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05e0105230 0x7f05e01056b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.873+0000 7f05e6e00640 1 --2- 172.21.15.16:0/1088798720 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e0104040 0x7f05e0104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.873+0000 7f05e6e00640 1 -- 172.21.15.16:0/1088798720 >> 172.21.15.16:0/1088798720 conn(0x7f05e00ff5d0 msgr2=0x7f05e01019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.873+0000 7f05e6e00640 1 -- 172.21.15.16:0/1088798720 shutdown_connections 2024-04-13T03:33:55.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.873+0000 7f05e6e00640 1 -- 172.21.15.16:0/1088798720 wait complete. 2024-04-13T03:33:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.874+0000 7f05e6e00640 1 Processor -- start 2024-04-13T03:33:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.874+0000 7f05e6e00640 1 -- start start 2024-04-13T03:33:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e6e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05e0104040 0x7f05e019a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e6e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e0105230 0x7f05e019a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:55.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e6e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05e0105bf0 0x7f05e01a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e6e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010fcd0 con 0x7f05e0105230 2024-04-13T03:33:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e6e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010fb50 con 0x7f05e0105bf0 2024-04-13T03:33:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e6e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010fe50 con 0x7f05e0104040 2024-04-13T03:33:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e0105230 0x7f05e019a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e6400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05e0105bf0 0x7f05e01a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e5000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e0105230 0x7f05e019a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59438/0 (socket says 172.21.15.16:59438) 2024-04-13T03:33:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e5000640 1 -- 172.21.15.16:0/161247136 learned_addr learned my addr 172.21.15.16:0/161247136 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:55.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.875+0000 7f05e5a00640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05e0104040 0x7f05e019a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05e5000640 1 -- 172.21.15.16:0/161247136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05e0104040 msgr2=0x7f05e019a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05e5000640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05e0104040 0x7f05e019a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05e5000640 1 -- 172.21.15.16:0/161247136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05e0105bf0 msgr2=0x7f05e01a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05e5000640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05e0105bf0 0x7f05e01a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05e5000640 1 -- 172.21.15.16:0/161247136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e01a1ef0 con 0x7f05e0105230 2024-04-13T03:33:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05e6400640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05e0105bf0 0x7f05e01a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05e5000640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e0105230 0x7f05e019a810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f05d40325f0 tx=0x7f05d4035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05de000640 1 -- 172.21.15.16:0/161247136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4039040 con 0x7f05e0105230 2024-04-13T03:33:55.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05de000640 1 -- 172.21.15.16:0/161247136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05d4044070 con 0x7f05e0105230 2024-04-13T03:33:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05de000640 1 -- 172.21.15.16:0/161247136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d40044d0 con 0x7f05e0105230 2024-04-13T03:33:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.876+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e01a2180 con 0x7f05e0105230 2024-04-13T03:33:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.877+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e01a2640 con 0x7f05e0105230 2024-04-13T03:33:55.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.878+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e010b300 con 0x7f05e0105230 2024-04-13T03:33:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.880+0000 7f05de000640 1 -- 172.21.15.16:0/161247136 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f05d4004670 con 0x7f05e0105230 2024-04-13T03:33:55.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.882+0000 7f05de000640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f05c80a3bb0 0x7f05c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:55.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.882+0000 7f05de000640 1 -- 172.21.15.16:0/161247136 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f05d40f2d70 con 0x7f05e0105230 2024-04-13T03:33:55.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.882+0000 7f05e5a00640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f05c80a3bb0 0x7f05c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:55.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.883+0000 7f05e5a00640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f05c80a3bb0 0x7f05c80a6070 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f05e0105090 tx=0x7f05d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:55.886+0000 7f05de000640 1 -- 172.21.15.16:0/161247136 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05d40bf380 con 0x7f05e0105230 2024-04-13T03:33:56.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.134+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05e0062bc0 con 0x7f05e0105230 2024-04-13T03:33:56.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.136+0000 7f05de000640 1 -- 172.21.15.16:0/161247136 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11934 (secure 0 0 0) 0x7f05d40bec80 con 0x7f05e0105230 2024-04-13T03:33:56.137 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:56.137 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:32:07.796132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24530,"mds_2":24523,"mds_3":24505,"mds_4":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24505,24500,24587,24523,24530]},"id":1}]} 2024-04-13T03:33:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 26 2024-04-13T03:33:56.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f05c80a3bb0 msgr2=0x7f05c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f05c80a3bb0 0x7f05c80a6070 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f05e0105090 tx=0x7f05d0005f90 comp rx=0 tx=0).stop 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e0105230 msgr2=0x7f05e019a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e0105230 0x7f05e019a810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f05d40325f0 tx=0x7f05d4035d10 comp rx=0 tx=0).stop 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 shutdown_connections 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f05c80a3bb0 0x7f05c80a6070 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05e0105bf0 0x7f05e01a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05e0105230 0x7f05e019a810 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 --2- 172.21.15.16:0/161247136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05e0104040 0x7f05e019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.138+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 >> 172.21.15.16:0/161247136 conn(0x7f05e00ff5d0 msgr2=0x7f05e01084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.139+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 shutdown_connections 2024-04-13T03:33:56.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.139+0000 7f05e6e00640 1 -- 172.21.15.16:0/161247136 wait complete. 2024-04-13T03:33:56.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.371+0000 7fd7c5400640 1 -- 172.21.15.16:0/4264532295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0074a20 msgr2=0x7fd7c0074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.371+0000 7fd7c5400640 1 --2- 172.21.15.16:0/4264532295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0074a20 0x7fd7c0074ea0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8009900 tx=0x7fd7a802bc40 comp rx=0 tx=0).stop 2024-04-13T03:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.371+0000 7fd7c5400640 1 -- 172.21.15.16:0/4264532295 shutdown_connections 2024-04-13T03:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.371+0000 7fd7c5400640 1 --2- 172.21.15.16:0/4264532295 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c0109d20 0x7fd7c010c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.371+0000 7fd7c5400640 1 --2- 172.21.15.16:0/4264532295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0074a20 0x7fd7c0074ea0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.371+0000 7fd7c5400640 1 --2- 172.21.15.16:0/4264532295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c0076000 0x7fd7c0074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.371+0000 7fd7c5400640 1 -- 172.21.15.16:0/4264532295 >> 172.21.15.16:0/4264532295 conn(0x7fd7c00fd3d0 msgr2=0x7fd7c00ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.371+0000 7fd7c5400640 1 -- 172.21.15.16:0/4264532295 shutdown_connections 2024-04-13T03:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.371+0000 7fd7c5400640 1 -- 172.21.15.16:0/4264532295 wait complete. 2024-04-13T03:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.373+0000 7fd7c5400640 1 Processor -- start 2024-04-13T03:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.373+0000 7fd7c5400640 1 -- start start 2024-04-13T03:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.373+0000 7fd7c5400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c0074a20 0x7fd7c0105130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.373+0000 7fd7c5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0076000 0x7fd7c0105670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.373+0000 7fd7c5400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c0109d20 0x7fd7c0101ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.373+0000 7fd7c5400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c010fd30 con 0x7fd7c0074a20 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.373+0000 7fd7c5400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c010fbb0 con 0x7fd7c0076000 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.373+0000 7fd7c5400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c010feb0 con 0x7fd7c0109d20 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7bfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c0074a20 0x7fd7c0105130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7b7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0076000 0x7fd7c0105670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7b7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0076000 0x7fd7c0105670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:35300/0 (socket says 172.21.15.16:35300) 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7b7e00640 1 -- 172.21.15.16:0/858971917 learned_addr learned my addr 172.21.15.16:0/858971917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7c4a00640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c0109d20 0x7fd7c0101ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7b7e00640 1 -- 172.21.15.16:0/858971917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c0109d20 msgr2=0x7fd7c0101ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7b7e00640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c0109d20 0x7fd7c0101ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7b7e00640 1 -- 172.21.15.16:0/858971917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c0074a20 msgr2=0x7fd7c0105130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7b7e00640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c0074a20 0x7fd7c0105130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7b7e00640 1 -- 172.21.15.16:0/858971917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c0102560 con 0x7fd7c0076000 2024-04-13T03:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.374+0000 7fd7b7e00640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0076000 0x7fd7c0105670 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd7a802c8f0 tx=0x7fd7a802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.375+0000 7fd7bd600640 1 -- 172.21.15.16:0/858971917 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8041070 con 0x7fd7c0076000 2024-04-13T03:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.375+0000 7fd7bd600640 1 -- 172.21.15.16:0/858971917 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a8004450 con 0x7fd7c0076000 2024-04-13T03:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.375+0000 7fd7bd600640 1 -- 172.21.15.16:0/858971917 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8034440 con 0x7fd7c0076000 2024-04-13T03:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.375+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c01027f0 con 0x7fd7c0076000 2024-04-13T03:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.375+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c01a6b00 con 0x7fd7c0076000 2024-04-13T03:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.376+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c0108a50 con 0x7fd7c0076000 2024-04-13T03:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.378+0000 7fd7bd600640 1 -- 172.21.15.16:0/858971917 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd7a8044070 con 0x7fd7c0076000 2024-04-13T03:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.379+0000 7fd7bd600640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd7a40a3d00 0x7fd7a40a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.380+0000 7fd7bd600640 1 -- 172.21.15.16:0/858971917 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd7a80f21e0 con 0x7fd7c0076000 2024-04-13T03:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.380+0000 7fd7bfe00640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd7a40a3d00 0x7fd7a40a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:56.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.381+0000 7fd7bfe00640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd7a40a3d00 0x7fd7a40a61c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009bd0 tx=0x7fd7b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.384+0000 7fd7bd600640 1 -- 172.21.15.16:0/858971917 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd7a80be140 con 0x7fd7c0076000 2024-04-13T03:33:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:56 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2138796587' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:33:56.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:56 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/161247136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.594+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7c0062bc0 con 0x7fd7c0076000 2024-04-13T03:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.597+0000 7fd7bd600640 1 -- 172.21.15.16:0/858971917 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24874 (secure 0 0 0) 0x7fd7a80c2440 con 0x7fd7c0076000 2024-04-13T03:33:56.598 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:56.598 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":94,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:33:42.153326+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.599+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd7a40a3d00 msgr2=0x7fd7a40a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.599+0000 7fd7c5400640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd7a40a3d00 0x7fd7a40a61c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009bd0 tx=0x7fd7b0005f90 comp rx=0 tx=0).stop 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0076000 msgr2=0x7fd7c0105670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0076000 0x7fd7c0105670 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd7a802c8f0 tx=0x7fd7a802c660 comp rx=0 tx=0).stop 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 shutdown_connections 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd7a40a3d00 0x7fd7a40a61c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7c0109d20 0x7fd7c0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7c0076000 0x7fd7c0105670 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 --2- 172.21.15.16:0/858971917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7c0074a20 0x7fd7c0105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 >> 172.21.15.16:0/858971917 conn(0x7fd7c00fd3d0 msgr2=0x7fd7c0105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 shutdown_connections 2024-04-13T03:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.600+0000 7fd7c5400640 1 -- 172.21.15.16:0/858971917 wait complete. 2024-04-13T03:33:56.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:56 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2138796587' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:33:56.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:56 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/161247136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:33:56.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:33:56.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:56 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2138796587' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:33:56.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:56 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/161247136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:33:56.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.836+0000 7f80cf200640 1 -- 172.21.15.16:0/2063009139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c8105230 msgr2=0x7f80c81056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.836+0000 7f80cf200640 1 --2- 172.21.15.16:0/2063009139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c8105230 0x7f80c81056b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f80b4009900 tx=0x7f80b402bc30 comp rx=0 tx=0).stop 2024-04-13T03:33:56.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.836+0000 7f80cf200640 1 -- 172.21.15.16:0/2063009139 shutdown_connections 2024-04-13T03:33:56.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.836+0000 7f80cf200640 1 --2- 172.21.15.16:0/2063009139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80c8105bf0 0x7f80c810c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.836+0000 7f80cf200640 1 --2- 172.21.15.16:0/2063009139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c8105230 0x7f80c81056b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.836+0000 7f80cf200640 1 --2- 172.21.15.16:0/2063009139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c8104040 0x7f80c8104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.837+0000 7f80cf200640 1 -- 172.21.15.16:0/2063009139 >> 172.21.15.16:0/2063009139 conn(0x7f80c80ff5d0 msgr2=0x7f80c81019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:56.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.837+0000 7f80cf200640 1 -- 172.21.15.16:0/2063009139 shutdown_connections 2024-04-13T03:33:56.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.837+0000 7f80cf200640 1 -- 172.21.15.16:0/2063009139 wait complete. 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.838+0000 7f80cf200640 1 Processor -- start 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.838+0000 7f80cf200640 1 -- start start 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.838+0000 7f80cf200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80c8104040 0x7f80c819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.838+0000 7f80cf200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c8105230 0x7f80c819a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.838+0000 7f80cf200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c8105bf0 0x7f80c81a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.838+0000 7f80cf200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80c810fcd0 con 0x7f80c8105230 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.838+0000 7f80cf200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f80c810fb50 con 0x7f80c8104040 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.838+0000 7f80cf200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f80c810fe50 con 0x7f80c8105bf0 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cd400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c8105230 0x7f80c819a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:56.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cde00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80c8104040 0x7f80c819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cde00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80c8104040 0x7f80c819a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:35314/0 (socket says 172.21.15.16:35314) 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cde00640 1 -- 172.21.15.16:0/1609709197 learned_addr learned my addr 172.21.15.16:0/1609709197 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80ce800640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c8105bf0 0x7f80c81a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cde00640 1 -- 172.21.15.16:0/1609709197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c8105bf0 msgr2=0x7f80c81a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cde00640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c8105bf0 0x7f80c81a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cde00640 1 -- 172.21.15.16:0/1609709197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c8105230 msgr2=0x7f80c819a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cde00640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c8105230 0x7f80c819a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cde00640 1 -- 172.21.15.16:0/1609709197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c81a1f00 con 0x7f80c8104040 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80cde00640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80c8104040 0x7f80c819a2e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f80bc00e9c0 tx=0x7f80bc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.839+0000 7f80c2a00640 1 -- 172.21.15.16:0/1609709197 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc00cd80 con 0x7f80c8104040 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.840+0000 7f80c2a00640 1 -- 172.21.15.16:0/1609709197 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80bc0035f0 con 0x7f80c8104040 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.840+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c81a21f0 con 0x7f80c8104040 2024-04-13T03:33:56.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.840+0000 7f80c2a00640 1 -- 172.21.15.16:0/1609709197 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc010690 con 0x7f80c8104040 2024-04-13T03:33:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.840+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c81a2710 con 0x7f80c8104040 2024-04-13T03:33:56.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.841+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c810b360 con 0x7f80c8104040 2024-04-13T03:33:56.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.843+0000 7f80c2a00640 1 -- 172.21.15.16:0/1609709197 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f80bc0037a0 con 0x7f80c8104040 2024-04-13T03:33:56.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.844+0000 7f80c2a00640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80b00a3bb0 0x7f80b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:56.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.844+0000 7f80c2a00640 1 -- 172.21.15.16:0/1609709197 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f80bc0d0cd0 con 0x7f80c8104040 2024-04-13T03:33:56.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.845+0000 7f80cd400640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80b00a3bb0 0x7f80b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:56.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.845+0000 7f80cd400640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80b00a3bb0 0x7f80b00a6070 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f80b402c7b0 tx=0x7f80b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:56.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:56.846+0000 7f80c2a00640 1 -- 172.21.15.16:0/1609709197 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80bc09d310 con 0x7f80c8104040 2024-04-13T03:33:57.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.095+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80c802d5e0 con 0x7f80c8104040 2024-04-13T03:33:57.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.097+0000 7f80c2a00640 1 -- 172.21.15.16:0/1609709197 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11934 (secure 0 0 0) 0x7f80bc09cc10 con 0x7f80c8104040 2024-04-13T03:33:57.098 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:33:57.098 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:32:07.796132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24530,"mds_2":24523,"mds_3":24505,"mds_4":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24505,24500,24587,24523,24530]},"id":1}]} 2024-04-13T03:33:57.099 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 26 2024-04-13T03:33:57.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80b00a3bb0 msgr2=0x7f80b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80b00a3bb0 0x7f80b00a6070 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f80b402c7b0 tx=0x7f80b4005fd0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80c8104040 msgr2=0x7f80c819a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80c8104040 0x7f80c819a2e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f80bc00e9c0 tx=0x7f80bc00ee80 comp rx=0 tx=0).stop 2024-04-13T03:33:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 shutdown_connections 2024-04-13T03:33:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f80b00a3bb0 0x7f80b00a6070 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80c8105bf0 0x7f80c81a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80c8105230 0x7f80c819a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 --2- 172.21.15.16:0/1609709197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80c8104040 0x7f80c819a2e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 >> 172.21.15.16:0/1609709197 conn(0x7f80c80ff5d0 msgr2=0x7f80c8108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 shutdown_connections 2024-04-13T03:33:57.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.100+0000 7f80cf200640 1 -- 172.21.15.16:0/1609709197 wait complete. 2024-04-13T03:33:57.126 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-13T03:31:37.109492+0000', 'modified': '2024-04-13T03:32:07.796132+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24587, 'mds_1': 24530, 'mds_2': 24523, 'mds_3': 24505, 'mds_4': 24500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24433': {'gid': 24433, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6833/3805092939', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6832', 'nonce': 3805092939}, {'type': 'v1', 'addr': '172.21.15.188:6833', 'nonce': 3805092939}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6833/3221738952', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6832', 'nonce': 3221738952}, {'type': 'v1', 'addr': '172.21.15.119:6833', 'nonce': 3221738952}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6835/1177424525', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 1177424525}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 1177424525}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6837/2782624805', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 2782624805}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 2782624805}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/566226134', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 566226134}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 566226134}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'e', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.119:6835/4186042513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 4186042513}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 4186042513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.188:6837/284865646', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 284865646}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 284865646}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.188:6839/57242391', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 57242391}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 57242391}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.119:6837/477256695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6836', 'nonce': 477256695}, {'type': 'v1', 'addr': '172.21.15.119:6837', 'nonce': 477256695}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/3722020849', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 3722020849}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 3722020849}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24587, 'qdb_cluster': [24505, 24500, 24587, 24523, 24530]} 2024-04-13T03:33:57.126 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-13T03:33:57.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-13T03:33:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.357+0000 7fadd6c00640 1 -- 172.21.15.16:0/2962585994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadd0102a50 msgr2=0x7fadd0109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.357+0000 7fadd6c00640 1 --2- 172.21.15.16:0/2962585994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadd0102a50 0x7fadd0109370 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009ca0 tx=0x7fadc802c100 comp rx=0 tx=0).stop 2024-04-13T03:33:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.357+0000 7fadd6c00640 1 -- 172.21.15.16:0/2962585994 shutdown_connections 2024-04-13T03:33:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.357+0000 7fadd6c00640 1 --2- 172.21.15.16:0/2962585994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadd0102a50 0x7fadd0109370 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.357+0000 7fadd6c00640 1 --2- 172.21.15.16:0/2962585994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd0102090 0x7fadd0102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.357+0000 7fadd6c00640 1 --2- 172.21.15.16:0/2962585994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadd0100eb0 0x7fadd01012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.358+0000 7fadd6c00640 1 -- 172.21.15.16:0/2962585994 >> 172.21.15.16:0/2962585994 conn(0x7fadd00fc470 msgr2=0x7fadd00fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.358+0000 7fadd6c00640 1 -- 172.21.15.16:0/2962585994 shutdown_connections 2024-04-13T03:33:57.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.358+0000 7fadd6c00640 1 -- 172.21.15.16:0/2962585994 wait complete. 2024-04-13T03:33:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.359+0000 7fadd6c00640 1 Processor -- start 2024-04-13T03:33:57.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.359+0000 7fadd6c00640 1 -- start start 2024-04-13T03:33:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadd6c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadd0100eb0 0x7fadd0104f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadd6c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadd0102090 0x7fadd0103570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadd6c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd0102a50 0x7fadd0103ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadd6c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fadd010cb80 con 0x7fadd0102a50 2024-04-13T03:33:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadd6c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fadd010ca00 con 0x7fadd0102090 2024-04-13T03:33:57.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadd6c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fadd010cd00 con 0x7fadd0100eb0 2024-04-13T03:33:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadd4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd0102a50 0x7fadd0103ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadd4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd0102a50 0x7fadd0103ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59504/0 (socket says 172.21.15.16:59504) 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadd4c00640 1 -- 172.21.15.16:0/413998226 learned_addr learned my addr 172.21.15.16:0/413998226 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadc7e00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadd0102090 0x7fadd0103570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.360+0000 7fadcfe00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadd0100eb0 0x7fadd0104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadd4c00640 1 -- 172.21.15.16:0/413998226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadd0100eb0 msgr2=0x7fadd0104f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadd4c00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadd0100eb0 0x7fadd0104f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadd4c00640 1 -- 172.21.15.16:0/413998226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadd0102090 msgr2=0x7fadd0103570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadd4c00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadd0102090 0x7fadd0103570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadd4c00640 1 -- 172.21.15.16:0/413998226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd0104340 con 0x7fadd0102a50 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadd4c00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd0102a50 0x7fadd0103ab0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fadc8037040 tx=0x7fadc8035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadcd600640 1 -- 172.21.15.16:0/413998226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8039040 con 0x7fadd0102a50 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadcd600640 1 -- 172.21.15.16:0/413998226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc8044070 con 0x7fadd0102a50 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadd6c00640 1 -- 172.21.15.16:0/413998226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd01a2390 con 0x7fadd0102a50 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.361+0000 7fadcd600640 1 -- 172.21.15.16:0/413998226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc80043a0 con 0x7fadd0102a50 2024-04-13T03:33:57.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.362+0000 7fadd6c00640 1 -- 172.21.15.16:0/413998226 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd01a28e0 con 0x7fadd0102a50 2024-04-13T03:33:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.365+0000 7fadd9ba4740 1 Processor -- start 2024-04-13T03:33:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.365+0000 7fadcd600640 1 -- 172.21.15.16:0/413998226 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fadc8004540 con 0x7fadd0102a50 2024-04-13T03:33:57.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.365+0000 7fadd9ba4740 1 -- start start 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 0x561865e7a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865e82560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561865de0990 con 0x561865e800d0 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561865d87530 con 0x561865e7a570 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561865ddc950 con 0x561865e7af60 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadcfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadcfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53310/0 (socket says 172.21.15.16:53310) 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadcfe00640 1 -- 172.21.15.16:0/3363885318 learned_addr learned my addr 172.21.15.16:0/3363885318 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865e82560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.366+0000 7fadc7e00640 1 --2- 172.21.15.16:0/3363885318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 0x561865e7a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadcfe00640 1 -- 172.21.15.16:0/3363885318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 msgr2=0x561865e7a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadcfe00640 1 --2- 172.21.15.16:0/3363885318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 0x561865e7a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadcfe00640 1 -- 172.21.15.16:0/3363885318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 msgr2=0x561865e82560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadcfe00640 1 --2- 172.21.15.16:0/3363885318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865e82560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadcfe00640 1 -- 172.21.15.16:0/3363885318 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561865af32d0 con 0x561865e7af60 2024-04-13T03:33:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadcd600640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fadb40a3bb0 0x7fadb40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadcfe00640 1 --2- 172.21.15.16:0/3363885318 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7fa60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009ca0 tx=0x7fadc0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5d23e8604c75c04 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadcd600640 1 -- 172.21.15.16:0/413998226 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fadc80f2e50 con 0x7fadd0102a50 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadc7e00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fadb40a3bb0 0x7fadb40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadc5200640 1 -- 172.21.15.16:0/3363885318 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0004220 con 0x561865e7af60 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadc5200640 1 -- 172.21.15.16:0/3363885318 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc00043c0 con 0x561865e7af60 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.367+0000 7fadc5200640 1 -- 172.21.15.16:0/3363885318 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0019930 con 0x561865e7af60 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadd9ba4740 1 -- 172.21.15.16:0/3363885318 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 msgr2=0x561865e7fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/3363885318 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7fa60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009ca0 tx=0x7fadc0011810 comp rx=0 tx=0).stop 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadc7e00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fadb40a3bb0 0x7fadb40a6070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x561865e7b7c0 tx=0x7fadb80096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadd9ba4740 1 -- 172.21.15.16:0/3363885318 shutdown_connections 2024-04-13T03:33:57.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/3363885318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865e82560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/3363885318 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7fa60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/3363885318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 0x561865e7a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadd9ba4740 1 -- 172.21.15.16:0/3363885318 >> 172.21.15.16:0/3363885318 conn(0x561865e75ae0 msgr2=0x561865e75ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadd9ba4740 1 -- 172.21.15.16:0/3363885318 shutdown_connections 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.368+0000 7fadd9ba4740 1 -- 172.21.15.16:0/3363885318 wait complete. 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.369+0000 7fadd9ba4740 1 -- start start 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.369+0000 7fadd9ba4740 1 Processor -- start 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.369+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e7a570 0x561865e84ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.369+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 0x561865e85000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.369+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e800d0 0x561865e8c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.369+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561865e83080 con 0x561865e7a570 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.369+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561865d9a8f0 con 0x561865e7af60 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561865e83200 con 0x561865e800d0 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadd9ba4740 10 client.0 _ll_register_callbacks cb 0x561865d9a2e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:57.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadcfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e7a570 0x561865e84ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadcfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e7a570 0x561865e84ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59524/0 (socket says 172.21.15.16:59524) 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadcfe00640 1 -- 172.21.15.16:0/2674249693 learned_addr learned my addr 172.21.15.16:0/2674249693 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadc7e00640 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 0x561865e85000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadd4c00640 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e800d0 0x561865e8c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadcfe00640 1 -- 172.21.15.16:0/2674249693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e800d0 msgr2=0x561865e8c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadcfe00640 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e800d0 0x561865e8c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadcfe00640 1 -- 172.21.15.16:0/2674249693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 msgr2=0x561865e85000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadcfe00640 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 0x561865e85000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.370+0000 7fadcfe00640 1 -- 172.21.15.16:0/2674249693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561865bbd6b0 con 0x561865e7a570 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.371+0000 7fadcfe00640 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e7a570 0x561865e84ac0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fadc0011d10 tx=0x7fadc0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d0e0062f0480e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.371+0000 7fadbd600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.371+0000 7fadbd600640 1 -- 172.21.15.16:0/2674249693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0028070 con 0x561865e7a570 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.371+0000 7fadbd600640 1 -- 172.21.15.16:0/2674249693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fadc0005040 con 0x561865e7a570 2024-04-13T03:33:57.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.371+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2674249693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadc00043c0 con 0x561865e7a570 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.371+0000 7fadbd600640 1 -- 172.21.15.16:0/2674249693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fadc0034050 con 0x561865e7a570 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.371+0000 7fadd9ba4740 10 client.15057 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.371+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2674249693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561865e8f3b0 con 0x561865e7a570 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.372+0000 7fadbd600640 1 -- 172.21.15.16:0/2674249693 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc00043c0 con 0x561865e7a570 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.372+0000 7fadbd600640 1 -- 172.21.15.16:0/2674249693 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fadc0004a80 con 0x561865e7a570 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.372+0000 7fadd9ba4740 10 client.15057 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.372+0000 7fadd9ba4740 10 client.15057 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.372+0000 7fadd9ba4740 20 client.15057 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.372+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x561865e91810 0x561865e92c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.372+0000 7fadd9ba4740 4 client.15057 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.372+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2674249693 --> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561865d86b40 con 0x561865e91810 2024-04-13T03:33:57.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.373+0000 7fadc7e00640 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x561865e91810 0x561865e92c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.373+0000 7fadc7e00640 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x561865e91810 0x561865e92c80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b64f469044091f77 server_cookie=3aabaea7d001340b in_seq=0 out_seq=0 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.373+0000 7fadbd600640 10 client.15057 ms_handle_connect on v2:172.21.15.119:6834/4186042513 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.374+0000 7fadbd600640 1 -- 172.21.15.16:0/2674249693 <== mds.4 v2:172.21.15.119:6834/4186042513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561865d86b40 con 0x561865e91810 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.374+0000 7fadbd600640 10 client.15057 handle_command_reply: tid=0 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.374+0000 7fadd9ba4740 1 client.15057 shutdown 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2674249693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e7a570 msgr2=0x561865e84ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e7a570 0x561865e84ac0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fadc0011d10 tx=0x7fadc0003820 comp rx=0 tx=0).stop 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2674249693 shutdown_connections 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x561865e91810 0x561865e92c80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e800d0 0x561865e8c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 0x561865e85000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2674249693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e7a570 0x561865e84ac0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2674249693 >> 172.21.15.16:0/2674249693 conn(0x561865e75ae0 msgr2=0x561865e74d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadbd600640 0 client.15057 ms_handle_reset on v2:172.21.15.119:6834/4186042513 2024-04-13T03:33:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2674249693 shutdown_connections 2024-04-13T03:33:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2674249693 wait complete. 2024-04-13T03:33:57.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.375+0000 7fadd9ba4740 20 client.15057 trim_cache size 0 max 16384 2024-04-13T03:33:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.395+0000 7fadd9ba4740 1 Processor -- start 2024-04-13T03:33:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.395+0000 7fadd9ba4740 1 -- start start 2024-04-13T03:33:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7a570 0x561865e824c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 0x561865e8bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865e835a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561865e90fc0 con 0x561865e800d0 2024-04-13T03:33:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561865e91690 con 0x561865e7af60 2024-04-13T03:33:57.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561865e91d60 con 0x561865e7a570 2024-04-13T03:33:57.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadd4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865e835a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadcfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7a570 0x561865e824c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadcfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7a570 0x561865e824c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53318/0 (socket says 172.21.15.16:53318) 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadcfe00640 1 -- 172.21.15.16:0/2640357008 learned_addr learned my addr 172.21.15.16:0/2640357008 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadc7e00640 1 --2- 172.21.15.16:0/2640357008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 0x561865e8bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.396+0000 7fadcfe00640 1 -- 172.21.15.16:0/2640357008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 msgr2=0x561865e8bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.397+0000 7fadcfe00640 1 --2- 172.21.15.16:0/2640357008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 0x561865e8bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.397+0000 7fadcfe00640 1 -- 172.21.15.16:0/2640357008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 msgr2=0x561865e835a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.397+0000 7fadcfe00640 1 --2- 172.21.15.16:0/2640357008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865e835a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.397+0000 7fadcfe00640 1 -- 172.21.15.16:0/2640357008 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561865e7f770 con 0x561865e7a570 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.397+0000 7fadd4c00640 1 --2- 172.21.15.16:0/2640357008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865e835a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.397+0000 7fadcfe00640 1 --2- 172.21.15.16:0/2640357008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7a570 0x561865e824c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fadc0004e50 tx=0x7fadc0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7316e553fbfa0494 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.397+0000 7fadbfe00640 1 -- 172.21.15.16:0/2640357008 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0028070 con 0x561865e7a570 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.397+0000 7fadbfe00640 1 -- 172.21.15.16:0/2640357008 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc0009e50 con 0x561865e7a570 2024-04-13T03:33:57.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.397+0000 7fadbfe00640 1 -- 172.21.15.16:0/2640357008 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc00055d0 con 0x561865e7a570 2024-04-13T03:33:57.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2640357008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7a570 msgr2=0x561865e824c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2640357008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7a570 0x561865e824c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fadc0004e50 tx=0x7fadc0011810 comp rx=0 tx=0).stop 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2640357008 shutdown_connections 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2640357008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865e835a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2640357008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7af60 0x561865e8bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/2640357008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7a570 0x561865e824c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2640357008 >> 172.21.15.16:0/2640357008 conn(0x561865e75ae0 msgr2=0x561865e795d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2640357008 shutdown_connections 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 -- 172.21.15.16:0/2640357008 wait complete. 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.398+0000 7fadd9ba4740 1 -- start start 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd9ba4740 1 Processor -- start 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 0x561865e7e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd9ba4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865ef57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561865e92b00 con 0x561865e800d0 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561865e92430 con 0x561865e7a570 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd9ba4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x561865e931d0 con 0x561865e7af60 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd9ba4740 10 client.0 _ll_register_callbacks cb 0x7fadc0004230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865ef57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.399+0000 7fadd4c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865ef57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59554/0 (socket says 172.21.15.16:59554) 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadd4c00640 1 -- 172.21.15.16:0/442643611 learned_addr learned my addr 172.21.15.16:0/442643611 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadc7e00640 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadcfe00640 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 0x561865e7e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadd4c00640 1 -- 172.21.15.16:0/442643611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 msgr2=0x561865e7e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadd4c00640 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadd4c00640 1 -- 172.21.15.16:0/442643611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 msgr2=0x561865e7e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadd4c00640 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 0x561865e7e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadd4c00640 1 -- 172.21.15.16:0/442643611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561865ef5cf0 con 0x561865e800d0 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadd4c00640 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865ef57b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fadc80bec90 tx=0x7fadc8032a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdeca2bd9db53fc server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadbd600640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadbd600640 1 -- 172.21.15.16:0/442643611 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc80d8060 con 0x561865e800d0 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadbd600640 1 -- 172.21.15.16:0/442643611 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fadc80f0630 con 0x561865e800d0 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.400+0000 7fadd9ba4740 1 -- 172.21.15.16:0/442643611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadc0009e50 con 0x561865e800d0 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadbd600640 1 -- 172.21.15.16:0/442643611 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fadc80bf400 con 0x561865e800d0 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadd9ba4740 10 client.15063 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadd9ba4740 1 -- 172.21.15.16:0/442643611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561865ef6eb0 con 0x561865e800d0 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadbd600640 1 -- 172.21.15.16:0/442643611 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc0009e50 con 0x561865e800d0 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadbd600640 1 -- 172.21.15.16:0/442643611 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fadc803c030 con 0x561865e800d0 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadd9ba4740 10 client.15063 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadd9ba4740 10 client.15063 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadd9ba4740 20 client.15063 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x561865efbb70 0x561865ef7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadd9ba4740 4 client.15063 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.401+0000 7fadd9ba4740 1 -- 172.21.15.16:0/442643611 --> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] -- command(tid 0: {"prefix": "status"}) -- 0x561865dee3b0 con 0x561865efbb70 2024-04-13T03:33:57.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.402+0000 7fadcfe00640 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x561865efbb70 0x561865ef7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.402+0000 7fadcfe00640 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x561865efbb70 0x561865ef7fb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=42496859b99f40f6 server_cookie=76a47f6e78c42362 in_seq=0 out_seq=0 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.402+0000 7fadbd600640 10 client.15063 ms_handle_connect on v2:172.21.15.119:6834/4186042513 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadbd600640 1 -- 172.21.15.16:0/442643611 <== mds.4 v2:172.21.15.119:6834/4186042513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x561865dee3b0 con 0x561865efbb70 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadbd600640 10 client.15063 handle_command_reply: tid=0 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 client.15063 shutdown 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 -- 172.21.15.16:0/442643611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 msgr2=0x561865ef57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865ef57b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fadc80bec90 tx=0x7fadc8032a70 comp rx=0 tx=0).stop 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 -- 172.21.15.16:0/442643611 shutdown_connections 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561865e800d0 0x561865ef57b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x561865efbb70 0x561865ef7fb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x561865e7af60 0x561865e7e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 --2- 172.21.15.16:0/442643611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561865e7a570 0x561865e7e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadbd600640 0 client.15063 ms_handle_reset on v2:172.21.15.119:6834/4186042513 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 -- 172.21.15.16:0/442643611 >> 172.21.15.16:0/442643611 conn(0x561865e75ae0 msgr2=0x561865e907c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 -- 172.21.15.16:0/442643611 shutdown_connections 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 1 -- 172.21.15.16:0/442643611 wait complete. 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.403+0000 7fadd9ba4740 20 client.15063 trim_cache size 0 max 16384 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-04-13T03:33:57.408 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24500, 2024-04-13T03:33:57.409 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:33:57.409 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:33:57.409 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:33:57.409 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 110.332775488, 2024-04-13T03:33:57.409 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:33:57.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:33:57.409 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:33:57.409 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 169.320773831 2024-04-13T03:33:57.409 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:33:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 -- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fadb40a3bb0 msgr2=0x7fadb40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fadb40a3bb0 0x7fadb40a6070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x561865e7b7c0 tx=0x7fadb80096c0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 -- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd0102a50 msgr2=0x7fadd0103ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd0102a50 0x7fadd0103ab0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fadc8037040 tx=0x7fadc8035d90 comp rx=0 tx=0).stop 2024-04-13T03:33:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 -- 172.21.15.16:0/413998226 shutdown_connections 2024-04-13T03:33:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fadb40a3bb0 0x7fadb40a6070 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fadd0102a50 0x7fadd0103ab0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadd0102090 0x7fadd0103570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 --2- 172.21.15.16:0/413998226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadd0100eb0 0x7fadd0104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 -- 172.21.15.16:0/413998226 >> 172.21.15.16:0/413998226 conn(0x7fadd00fc470 msgr2=0x7fadd0069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 -- 172.21.15.16:0/413998226 shutdown_connections 2024-04-13T03:33:57.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.404+0000 7fadd6c00640 1 -- 172.21.15.16:0/413998226 wait complete. 2024-04-13T03:33:57.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:57 smithi016 ceph-mon[31772]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-13T03:33:57.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:57 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/858971917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:33:57.421 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:57 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1609709197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:33:57.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-13T03:33:57.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:57 smithi119 ceph-mon[35835]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-13T03:33:57.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:57 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/858971917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:33:57.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:57 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1609709197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:33:57.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:57 smithi188 ceph-mon[36398]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-13T03:33:57.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:57 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/858971917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:33:57.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:57 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1609709197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:33:57.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.643+0000 7f39be800640 1 -- 172.21.15.16:0/2210219249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8077bc0 msgr2=0x7f39b8078020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.643+0000 7f39be800640 1 --2- 172.21.15.16:0/2210219249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8077bc0 0x7f39b8078020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009900 tx=0x7f39a402bc30 comp rx=0 tx=0).stop 2024-04-13T03:33:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.643+0000 7f39be800640 1 -- 172.21.15.16:0/2210219249 shutdown_connections 2024-04-13T03:33:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.643+0000 7f39be800640 1 --2- 172.21.15.16:0/2210219249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39b8078560 0x7f39b8078e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.643+0000 7f39be800640 1 --2- 172.21.15.16:0/2210219249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8077bc0 0x7f39b8078020 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.643+0000 7f39be800640 1 --2- 172.21.15.16:0/2210219249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39b8076980 0x7f39b8076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.643+0000 7f39be800640 1 -- 172.21.15.16:0/2210219249 >> 172.21.15.16:0/2210219249 conn(0x7f39b80ff770 msgr2=0x7f39b8101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.643+0000 7f39be800640 1 -- 172.21.15.16:0/2210219249 shutdown_connections 2024-04-13T03:33:57.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.643+0000 7f39be800640 1 -- 172.21.15.16:0/2210219249 wait complete. 2024-04-13T03:33:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.644+0000 7f39be800640 1 Processor -- start 2024-04-13T03:33:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.644+0000 7f39be800640 1 -- start start 2024-04-13T03:33:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39be800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8076980 0x7f39b819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39be800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39b8077bc0 0x7f39b819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39be800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39b8078560 0x7f39b81a5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39be800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f39b81142b0 con 0x7f39b8076980 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39be800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f39b8114130 con 0x7f39b8078560 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39be800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f39b8114430 con 0x7f39b8077bc0 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39b7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8076980 0x7f39b819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39b7400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8076980 0x7f39b819e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59560/0 (socket says 172.21.15.16:59560) 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39b7400640 1 -- 172.21.15.16:0/394782361 learned_addr learned my addr 172.21.15.16:0/394782361 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39b7e00640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39b8078560 0x7f39b81a5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.645+0000 7f39b6a00640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39b8077bc0 0x7f39b819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39b7400640 1 -- 172.21.15.16:0/394782361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39b8077bc0 msgr2=0x7f39b819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39b7400640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39b8077bc0 0x7f39b819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39b7400640 1 -- 172.21.15.16:0/394782361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39b8078560 msgr2=0x7f39b81a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39b7400640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39b8078560 0x7f39b81a5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39b7400640 1 -- 172.21.15.16:0/394782361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b81a6570 con 0x7f39b8076980 2024-04-13T03:33:57.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39b7400640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8076980 0x7f39b819e950 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f39a800e9f0 tx=0x7f39a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39afe00640 1 -- 172.21.15.16:0/394782361 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a800ccc0 con 0x7f39b8076980 2024-04-13T03:33:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39afe00640 1 -- 172.21.15.16:0/394782361 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39a8010910 con 0x7f39b8076980 2024-04-13T03:33:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39be800640 1 -- 172.21.15.16:0/394782361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b81a6860 con 0x7f39b8076980 2024-04-13T03:33:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39afe00640 1 -- 172.21.15.16:0/394782361 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a80188a0 con 0x7f39b8076980 2024-04-13T03:33:57.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.646+0000 7f39be800640 1 -- 172.21.15.16:0/394782361 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b81a6dd0 con 0x7f39b8076980 2024-04-13T03:33:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.650+0000 7f39c1770740 1 Processor -- start 2024-04-13T03:33:57.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.651+0000 7f39c1770740 1 -- start start 2024-04-13T03:33:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.651+0000 7f39c1770740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28774990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.651+0000 7f39c1770740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be28779a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.651+0000 7f39c1770740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be2877a0d0 0x55be2877c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.651+0000 7f39c1770740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be286da990 con 0x55be2877a0d0 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.651+0000 7f39c1770740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55be28681530 con 0x55be28774f60 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.651+0000 7f39c1770740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55be286d6950 con 0x55be28774570 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.651+0000 7f39afe00640 1 -- 172.21.15.16:0/394782361 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f39a8010ac0 con 0x7f39b8076980 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28774990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be2877a0d0 0x55be2877c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be2877a0d0 0x55be2877c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59562/0 (socket says 172.21.15.16:59562) 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b6a00640 1 -- 172.21.15.16:0/4186309900 learned_addr learned my addr 172.21.15.16:0/4186309900 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28774990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53376/0 (socket says 172.21.15.16:53376) 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b7400640 1 --2- 172.21.15.16:0/4186309900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be28779a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b7e00640 1 -- 172.21.15.16:0/4186309900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 msgr2=0x55be28779a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4186309900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be28779a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b7e00640 1 -- 172.21.15.16:0/4186309900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be2877a0d0 msgr2=0x55be2877c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4186309900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be2877a0d0 0x55be2877c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b7e00640 1 -- 172.21.15.16:0/4186309900 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be283ed2d0 con 0x55be28774570 2024-04-13T03:33:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.652+0000 7f39b6a00640 1 --2- 172.21.15.16:0/4186309900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be2877a0d0 0x55be2877c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:57.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.653+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4186309900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28774990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009f10 tx=0x7f39b0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5a946ec823c3cfb server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.653+0000 7f39acc00640 1 -- 172.21.15.16:0/4186309900 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0004d20 con 0x55be28774570 2024-04-13T03:33:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.653+0000 7f39acc00640 1 -- 172.21.15.16:0/4186309900 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39b0004030 con 0x55be28774570 2024-04-13T03:33:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.653+0000 7f39acc00640 1 -- 172.21.15.16:0/4186309900 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0005260 con 0x55be28774570 2024-04-13T03:33:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.653+0000 7f39afe00640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f399c0a3c30 0x7f399c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39afe00640 1 -- 172.21.15.16:0/394782361 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f39a80d0bc0 con 0x7f39b8076980 2024-04-13T03:33:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39b7400640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f399c0a3c30 0x7f399c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39c1770740 1 -- 172.21.15.16:0/4186309900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 msgr2=0x55be28774990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39c1770740 1 --2- 172.21.15.16:0/4186309900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28774990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009f10 tx=0x7f39b0011810 comp rx=0 tx=0).stop 2024-04-13T03:33:57.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39c1770740 1 -- 172.21.15.16:0/4186309900 shutdown_connections 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39c1770740 1 --2- 172.21.15.16:0/4186309900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be2877a0d0 0x55be2877c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39c1770740 1 --2- 172.21.15.16:0/4186309900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be28779a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39c1770740 1 --2- 172.21.15.16:0/4186309900 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28774990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39c1770740 1 -- 172.21.15.16:0/4186309900 >> 172.21.15.16:0/4186309900 conn(0x55be2876fae0 msgr2=0x55be2876fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39c1770740 1 -- 172.21.15.16:0/4186309900 shutdown_connections 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39c1770740 1 -- 172.21.15.16:0/4186309900 wait complete. 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.654+0000 7f39b7400640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f399c0a3c30 0x7f399c0a60f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f39a8098ed0 tx=0x7f39a80d4600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.655+0000 7f39c1770740 1 -- start start 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.655+0000 7f39c1770740 1 Processor -- start 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.655+0000 7f39c1770740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774570 0x55be2877eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.655+0000 7f39c1770740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be2877f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.655+0000 7f39c1770740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be2877a0d0 0x55be287860b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.655+0000 7f39c1770740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be2877d080 con 0x55be28774570 2024-04-13T03:33:57.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.655+0000 7f39c1770740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55be286948f0 con 0x55be28774f60 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.655+0000 7f39c1770740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55be2877d200 con 0x55be2877a0d0 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.655+0000 7f39c1770740 10 client.0 _ll_register_callbacks cb 0x55be286942e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774570 0x55be2877eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be2877f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be2877f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:35414/0 (socket says 172.21.15.16:35414) 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b6a00640 1 -- 172.21.15.16:0/1520573431 learned_addr learned my addr 172.21.15.16:0/1520573431 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b7e00640 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be2877a0d0 0x55be287860b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b6a00640 1 -- 172.21.15.16:0/1520573431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be2877a0d0 msgr2=0x55be287860b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b6a00640 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be2877a0d0 0x55be287860b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b6a00640 1 -- 172.21.15.16:0/1520573431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774570 msgr2=0x55be2877eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b6a00640 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774570 0x55be2877eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b6a00640 1 -- 172.21.15.16:0/1520573431 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be284b76b0 con 0x55be28774f60 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b7e00640 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774570 0x55be2877eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39b6a00640 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be2877f070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009230 tx=0x7f39a4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e174a5a1dc82e2e server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.656+0000 7f39a1600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-13T03:33:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.657+0000 7f39a1600640 1 -- 172.21.15.16:0/1520573431 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4005d50 con 0x55be28774f60 2024-04-13T03:33:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.657+0000 7f39c1770740 1 -- 172.21.15.16:0/1520573431 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39b0004030 con 0x55be28774f60 2024-04-13T03:33:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.657+0000 7f39a1600640 1 -- 172.21.15.16:0/1520573431 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f39a4005230 con 0x55be28774f60 2024-04-13T03:33:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.657+0000 7f39a1600640 1 -- 172.21.15.16:0/1520573431 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f39a404e050 con 0x55be28774f60 2024-04-13T03:33:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.657+0000 7f39c1770740 10 client.25181 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:57.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.657+0000 7f39c1770740 1 -- 172.21.15.16:0/1520573431 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55be287894e0 con 0x55be28774f60 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.657+0000 7f39a1600640 1 -- 172.21.15.16:0/1520573431 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39b0004030 con 0x55be28774f60 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.658+0000 7f39a1600640 1 -- 172.21.15.16:0/1520573431 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f39a4003890 con 0x55be28774f60 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.658+0000 7f39c1770740 10 client.25181 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.658+0000 7f39c1770740 10 client.25181 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.658+0000 7f39c1770740 20 client.25181 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.658+0000 7f39c1770740 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x55be2878bac0 0x55be2878cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.658+0000 7f39c1770740 4 client.25181 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.658+0000 7f39c1770740 1 -- 172.21.15.16:0/1520573431 --> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be28680b40 con 0x55be2878bac0 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.658+0000 7f39b7e00640 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x55be2878bac0 0x55be2878cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.659+0000 7f39b7e00640 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x55be2878bac0 0x55be2878cf30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6ff6b79b9d47608a server_cookie=5f2da4cd5e883421 in_seq=0 out_seq=0 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.659+0000 7f39a1600640 10 client.25181 ms_handle_connect on v2:172.21.15.188:6836/284865646 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39a1600640 1 -- 172.21.15.16:0/1520573431 <== mds.3 v2:172.21.15.188:6836/284865646 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be28680b40 con 0x55be2878bac0 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39a1600640 10 client.25181 handle_command_reply: tid=0 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39c1770740 1 client.25181 shutdown 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39c1770740 1 -- 172.21.15.16:0/1520573431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 msgr2=0x55be2877f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39c1770740 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be2877f070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009230 tx=0x7f39a4004270 comp rx=0 tx=0).stop 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39c1770740 1 -- 172.21.15.16:0/1520573431 shutdown_connections 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39c1770740 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x55be2878bac0 0x55be2878cf30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39c1770740 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be2877a0d0 0x55be287860b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39c1770740 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be28774f60 0x55be2877f070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39c1770740 1 --2- 172.21.15.16:0/1520573431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774570 0x55be2877eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39a1600640 0 client.25181 ms_handle_reset on v2:172.21.15.188:6836/284865646 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.660+0000 7f39c1770740 1 -- 172.21.15.16:0/1520573431 >> 172.21.15.16:0/1520573431 conn(0x55be2876fae0 msgr2=0x55be2876ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.661+0000 7f39c1770740 1 -- 172.21.15.16:0/1520573431 shutdown_connections 2024-04-13T03:33:57.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.661+0000 7f39c1770740 1 -- 172.21.15.16:0/1520573431 wait complete. 2024-04-13T03:33:57.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.661+0000 7f39c1770740 20 client.25181 trim_cache size 0 max 16384 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.680+0000 7f39c1770740 1 Processor -- start 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39c1770740 1 -- start start 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39c1770740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be2877d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39c1770740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be28786070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39c1770740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be2876fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39c1770740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be2878b270 con 0x55be28774f60 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39c1770740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55be2878b940 con 0x55be2877a0d0 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39c1770740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55be2878c010 con 0x55be28774570 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39b6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be28786070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39b6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be28786070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59588/0 (socket says 172.21.15.16:59588) 2024-04-13T03:33:57.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39b6a00640 1 -- 172.21.15.16:0/4118620875 learned_addr learned my addr 172.21.15.16:0/4118620875 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.681+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be2877d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be2876fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39b6a00640 1 -- 172.21.15.16:0/4118620875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 msgr2=0x55be2877d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39b6a00640 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be2877d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39b6a00640 1 -- 172.21.15.16:0/4118620875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 msgr2=0x55be2876fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39b6a00640 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be2876fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39b6a00640 1 -- 172.21.15.16:0/4118620875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be28779e20 con 0x55be28774f60 2024-04-13T03:33:57.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be2876fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-13T03:33:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39b6a00640 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be28786070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f39a402c710 tx=0x7f39a402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968d773ee8ec1b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39a3e00640 1 -- 172.21.15.16:0/4118620875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4005d50 con 0x55be28774f60 2024-04-13T03:33:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.682+0000 7f39a3e00640 1 -- 172.21.15.16:0/4118620875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39a4041070 con 0x55be28774f60 2024-04-13T03:33:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39a3e00640 1 -- 172.21.15.16:0/4118620875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a40346a0 con 0x55be28774f60 2024-04-13T03:33:57.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39c1770740 1 -- 172.21.15.16:0/4118620875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 msgr2=0x55be28786070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39c1770740 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be28786070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f39a402c710 tx=0x7f39a402bc30 comp rx=0 tx=0).stop 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39c1770740 1 -- 172.21.15.16:0/4118620875 shutdown_connections 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39c1770740 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be2876fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39c1770740 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be28786070 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39c1770740 1 --2- 172.21.15.16:0/4118620875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be2877d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39c1770740 1 -- 172.21.15.16:0/4118620875 >> 172.21.15.16:0/4118620875 conn(0x55be2876fae0 msgr2=0x55be2876ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39c1770740 1 -- 172.21.15.16:0/4118620875 shutdown_connections 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.683+0000 7f39c1770740 1 -- 172.21.15.16:0/4118620875 wait complete. 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.684+0000 7f39c1770740 1 -- start start 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.684+0000 7f39c1770740 1 Processor -- start 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.684+0000 7f39c1770740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28783fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.684+0000 7f39c1770740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be287eda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.684+0000 7f39c1770740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be287f0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.684+0000 7f39c1770740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55be2878cdb0 con 0x55be28774f60 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.684+0000 7f39c1770740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55be2878c6e0 con 0x55be2877a0d0 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.684+0000 7f39c1770740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55be2878d480 con 0x55be28774570 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.684+0000 7f39c1770740 10 client.0 _ll_register_callbacks cb 0x7f39a402c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28783fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28783fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53400/0 (socket says 172.21.15.16:53400) 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b7e00640 1 -- 172.21.15.16:0/4122062311 learned_addr learned my addr 172.21.15.16:0/4122062311 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be287f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b6a00640 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be287eda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b7e00640 1 -- 172.21.15.16:0/4122062311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 msgr2=0x55be28783fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28783fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b7e00640 1 -- 172.21.15.16:0/4122062311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 msgr2=0x55be287eda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be287eda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b7e00640 1 -- 172.21.15.16:0/4122062311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be287f0850 con 0x55be2877a0d0 2024-04-13T03:33:57.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.685+0000 7f39b6a00640 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be287eda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:57.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.686+0000 7f39b7e00640 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be287f0020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009d50 tx=0x7f39b0004810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28070bfb2d2b93e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.686+0000 7f39a1600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-13T03:33:57.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.686+0000 7f39a1600640 1 -- 172.21.15.16:0/4122062311 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b00044f0 con 0x55be2877a0d0 2024-04-13T03:33:57.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.686+0000 7f39c1770740 1 -- 172.21.15.16:0/4122062311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39a4041070 con 0x55be2877a0d0 2024-04-13T03:33:57.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.686+0000 7f39a1600640 1 -- 172.21.15.16:0/4122062311 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f39b0004880 con 0x55be2877a0d0 2024-04-13T03:33:57.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.686+0000 7f39a1600640 1 -- 172.21.15.16:0/4122062311 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f39b0004aa0 con 0x55be2877a0d0 2024-04-13T03:33:57.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.686+0000 7f39c1770740 10 client.25184 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.686+0000 7f39c1770740 1 -- 172.21.15.16:0/4122062311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55be287f1b30 con 0x55be2877a0d0 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.687+0000 7f39a1600640 1 -- 172.21.15.16:0/4122062311 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39a4041070 con 0x55be2877a0d0 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.687+0000 7f39a1600640 1 -- 172.21.15.16:0/4122062311 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f39b0024040 con 0x55be2877a0d0 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.687+0000 7f39c1770740 10 client.25184 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.687+0000 7f39c1770740 10 client.25184 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.687+0000 7f39c1770740 20 client.25184 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.687+0000 7f39c1770740 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x55be287f6520 0x55be287f5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.687+0000 7f39c1770740 4 client.25184 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.687+0000 7f39c1770740 1 -- 172.21.15.16:0/4122062311 --> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] -- command(tid 0: {"prefix": "status"}) -- 0x55be286e83b0 con 0x55be287f6520 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.687+0000 7f39b6a00640 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x55be287f6520 0x55be287f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39b6a00640 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x55be287f6520 0x55be287f5230 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5b3c582e4092ad20 server_cookie=b975edc7f4c6df3d in_seq=0 out_seq=0 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39a1600640 10 client.25184 ms_handle_connect on v2:172.21.15.188:6836/284865646 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39a1600640 1 -- 172.21.15.16:0/4122062311 <== mds.3 v2:172.21.15.188:6836/284865646 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55be286e83b0 con 0x55be287f6520 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39a1600640 10 client.25184 handle_command_reply: tid=0 2024-04-13T03:33:57.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 client.25184 shutdown 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 -- 172.21.15.16:0/4122062311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 msgr2=0x55be287f0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be287f0020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009d50 tx=0x7f39b0004810 comp rx=0 tx=0).stop 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 -- 172.21.15.16:0/4122062311 shutdown_connections 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x55be287f6520 0x55be287f5230 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55be2877a0d0 0x55be287f0020 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55be28774f60 0x55be287eda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 --2- 172.21.15.16:0/4122062311 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55be28774570 0x55be28783fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39a1600640 0 client.25184 ms_handle_reset on v2:172.21.15.188:6836/284865646 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 -- 172.21.15.16:0/4122062311 >> 172.21.15.16:0/4122062311 conn(0x55be2876fae0 msgr2=0x55be2877d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 -- 172.21.15.16:0/4122062311 shutdown_connections 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.688+0000 7f39c1770740 1 -- 172.21.15.16:0/4122062311 wait complete. 2024-04-13T03:33:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39c1770740 20 client.25184 trim_cache size 0 max 16384 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24505, 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 111.630970335, 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 163.802885226 2024-04-13T03:33:57.694 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:33:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 -- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f399c0a3c30 msgr2=0x7f399c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f399c0a3c30 0x7f399c0a60f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f39a8098ed0 tx=0x7f39a80d4600 comp rx=0 tx=0).stop 2024-04-13T03:33:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 -- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8076980 msgr2=0x7f39b819e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8076980 0x7f39b819e950 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f39a800e9f0 tx=0x7f39a800edf0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 -- 172.21.15.16:0/394782361 shutdown_connections 2024-04-13T03:33:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f399c0a3c30 0x7f399c0a60f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39b8078560 0x7f39b81a5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39b8077bc0 0x7f39b819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 --2- 172.21.15.16:0/394782361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f39b8076980 0x7f39b819e950 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.689+0000 7f39be800640 1 -- 172.21.15.16:0/394782361 >> 172.21.15.16:0/394782361 conn(0x7f39b80ff770 msgr2=0x7f39b8100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.690+0000 7f39be800640 1 -- 172.21.15.16:0/394782361 shutdown_connections 2024-04-13T03:33:57.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.690+0000 7f39be800640 1 -- 172.21.15.16:0/394782361 wait complete. 2024-04-13T03:33:57.711 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-13T03:33:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.932+0000 7f133ae00640 1 -- 172.21.15.16:0/3429947375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 msgr2=0x7f133410a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.932+0000 7f133ae00640 1 --2- 172.21.15.16:0/3429947375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 0x7f133410a3c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f132c009cd0 tx=0x7f132c02c0e0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.932+0000 7f133ae00640 1 -- 172.21.15.16:0/3429947375 shutdown_connections 2024-04-13T03:33:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.932+0000 7f133ae00640 1 --2- 172.21.15.16:0/3429947375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 0x7f133410a3c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.932+0000 7f133ae00640 1 --2- 172.21.15.16:0/3429947375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1334103170 0x7f13341035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.932+0000 7f133ae00640 1 --2- 172.21.15.16:0/3429947375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13341001a0 0x7f13341005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.932+0000 7f133ae00640 1 -- 172.21.15.16:0/3429947375 >> 172.21.15.16:0/3429947375 conn(0x7f13340fbaf0 msgr2=0x7f13340fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.932+0000 7f133ae00640 1 -- 172.21.15.16:0/3429947375 shutdown_connections 2024-04-13T03:33:57.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.932+0000 7f133ae00640 1 -- 172.21.15.16:0/3429947375 wait complete. 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.933+0000 7f133ae00640 1 Processor -- start 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.933+0000 7f133ae00640 1 -- start start 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133ae00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13341001a0 0x7f133419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133ae00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1334103170 0x7f133419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133ae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 0x7f13341a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133ae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f133410db60 con 0x7f1334103b30 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133ae00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f133410d9e0 con 0x7f1334103170 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133ae00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f133410dce0 con 0x7f13341001a0 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 0x7f13341a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f1339000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1334103170 0x7f133419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 0x7f13341a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59602/0 (socket says 172.21.15.16:59602) 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133a400640 1 -- 172.21.15.16:0/235463428 learned_addr learned my addr 172.21.15.16:0/235463428 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f1339a00640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13341001a0 0x7f133419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133a400640 1 -- 172.21.15.16:0/235463428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13341001a0 msgr2=0x7f133419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133a400640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13341001a0 0x7f133419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.934+0000 7f133a400640 1 -- 172.21.15.16:0/235463428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1334103170 msgr2=0x7f133419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.935+0000 7f133a400640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1334103170 0x7f133419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.935+0000 7f133a400640 1 -- 172.21.15.16:0/235463428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13341a1fe0 con 0x7f1334103b30 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.935+0000 7f133a400640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 0x7f13341a1970 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f132c0325d0 tx=0x7f132c035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.935+0000 7f1332000640 1 -- 172.21.15.16:0/235463428 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c039040 con 0x7f1334103b30 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.935+0000 7f1332000640 1 -- 172.21.15.16:0/235463428 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f132c044070 con 0x7f1334103b30 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.935+0000 7f1332000640 1 -- 172.21.15.16:0/235463428 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c0044a0 con 0x7f1334103b30 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.935+0000 7f133ae00640 1 -- 172.21.15.16:0/235463428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13341a2270 con 0x7f1334103b30 2024-04-13T03:33:57.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.935+0000 7f133ae00640 1 -- 172.21.15.16:0/235463428 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13341a2710 con 0x7f1334103b30 2024-04-13T03:33:57.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.939+0000 7f1332000640 1 -- 172.21.15.16:0/235463428 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f132c004640 con 0x7f1334103b30 2024-04-13T03:33:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.939+0000 7f133f248740 1 Processor -- start 2024-04-13T03:33:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.939+0000 7f133f248740 1 -- start start 2024-04-13T03:33:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.939+0000 7f133f248740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.939+0000 7f133f248740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9ac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.939+0000 7f133f248740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9acb20 0x55e6cc9aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.939+0000 7f133f248740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc904990 con 0x55e6cc9a79b0 2024-04-13T03:33:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.939+0000 7f133f248740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc8ab530 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.939+0000 7f133f248740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc900950 con 0x55e6cc9acb20 2024-04-13T03:33:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9ac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9ac4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59616/0 (socket says 172.21.15.16:59616) 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339a00640 1 -- 172.21.15.16:0/2197320406 learned_addr learned my addr 172.21.15.16:0/2197320406 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339000640 1 --2- 172.21.15.16:0/2197320406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339000640 1 --2- 172.21.15.16:0/2197320406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9acb20 0x55e6cc9aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339a00640 1 -- 172.21.15.16:0/2197320406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9acb20 msgr2=0x55e6cc9aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339a00640 1 --2- 172.21.15.16:0/2197320406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9acb20 0x55e6cc9aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339a00640 1 -- 172.21.15.16:0/2197320406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a6fc0 msgr2=0x55e6cc9a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339a00640 1 --2- 172.21.15.16:0/2197320406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9a73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339a00640 1 -- 172.21.15.16:0/2197320406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6cc6172d0 con 0x55e6cc9a79b0 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.940+0000 7f1339a00640 1 --2- 172.21.15.16:0/2197320406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9ac4b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1328009ca0 tx=0x7f1328011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed49494597b1841f server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.941+0000 7f1332000640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f131c0a3c30 0x7f131c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.941+0000 7f1326a00640 1 -- 172.21.15.16:0/2197320406 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1328004220 con 0x55e6cc9a79b0 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.941+0000 7f1332000640 1 -- 172.21.15.16:0/235463428 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f132c0f2ca0 con 0x7f1334103b30 2024-04-13T03:33:57.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.941+0000 7f1326a00640 1 -- 172.21.15.16:0/2197320406 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13280043c0 con 0x55e6cc9a79b0 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.941+0000 7f1326a00640 1 -- 172.21.15.16:0/2197320406 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1328019890 con 0x55e6cc9a79b0 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.941+0000 7f1339000640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f131c0a3c30 0x7f131c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 -- 172.21.15.16:0/2197320406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a79b0 msgr2=0x55e6cc9ac4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f1339000640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f131c0a3c30 0x7f131c0a60f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55e6cc9a8210 tx=0x7f1320009770 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 --2- 172.21.15.16:0/2197320406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9ac4b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1328009ca0 tx=0x7f1328011810 comp rx=0 tx=0).stop 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 -- 172.21.15.16:0/2197320406 shutdown_connections 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 --2- 172.21.15.16:0/2197320406 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9acb20 0x55e6cc9aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 --2- 172.21.15.16:0/2197320406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9ac4b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 --2- 172.21.15.16:0/2197320406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 -- 172.21.15.16:0/2197320406 >> 172.21.15.16:0/2197320406 conn(0x55e6cc9a2530 msgr2=0x55e6cc9a2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 -- 172.21.15.16:0/2197320406 shutdown_connections 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 -- 172.21.15.16:0/2197320406 wait complete. 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.942+0000 7f133f248740 1 -- start start 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.943+0000 7f133f248740 1 Processor -- start 2024-04-13T03:33:57.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.943+0000 7f133f248740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.943+0000 7f133f248740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.943+0000 7f133f248740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9acb20 0x55e6cc9b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.943+0000 7f133f248740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc9afad0 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.943+0000 7f133f248740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc8be8f0 con 0x55e6cc9acb20 2024-04-13T03:33:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.943+0000 7f133f248740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc9afc50 con 0x55e6cc9a79b0 2024-04-13T03:33:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.943+0000 7f133f248740 10 client.0 _ll_register_callbacks cb 0x55e6cc8be2e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.943+0000 7f1339a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f1339000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f1339a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59622/0 (socket says 172.21.15.16:59622) 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f1339a00640 1 -- 172.21.15.16:0/1470243338 learned_addr learned my addr 172.21.15.16:0/1470243338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f133a400640 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9acb20 0x55e6cc9b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f1339a00640 1 -- 172.21.15.16:0/1470243338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a79b0 msgr2=0x55e6cc9b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f1339a00640 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f1339a00640 1 -- 172.21.15.16:0/1470243338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9acb20 msgr2=0x55e6cc9b8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f1339a00640 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9acb20 0x55e6cc9b8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f1339a00640 1 -- 172.21.15.16:0/1470243338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6cc6e16b0 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f133a400640 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9acb20 0x55e6cc9b8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f1339a00640 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b14d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1328009f50 tx=0x7f1328003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6961dda08ab4e63c server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.944+0000 7f131b400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.945+0000 7f131b400640 1 -- 172.21.15.16:0/1470243338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1328028070 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.945+0000 7f131b400640 1 -- 172.21.15.16:0/1470243338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1328003800 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.945+0000 7f133f248740 1 -- 172.21.15.16:0/1470243338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13280043c0 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.945+0000 7f131b400640 1 -- 172.21.15.16:0/1470243338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1328034050 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.945+0000 7f133f248740 10 client.15093 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.945+0000 7f133f248740 1 -- 172.21.15.16:0/1470243338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e6cc9bbdc0 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.946+0000 7f131b400640 1 -- 172.21.15.16:0/1470243338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13280043c0 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.946+0000 7f131b400640 1 -- 172.21.15.16:0/1470243338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f1328004a40 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.946+0000 7f133f248740 10 client.15093 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.946+0000 7f133f248740 10 client.15093 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.946+0000 7f133f248740 20 client.15093 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.946+0000 7f133f248740 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x55e6cc9be220 0x55e6cc9bf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.946+0000 7f133f248740 4 client.15093 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.946+0000 7f133f248740 1 -- 172.21.15.16:0/1470243338 --> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6cc8aab40 con 0x55e6cc9be220 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.947+0000 7f1339000640 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x55e6cc9be220 0x55e6cc9bf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.947+0000 7f1339000640 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x55e6cc9be220 0x55e6cc9bf690 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=de6a3ec128d69d11 server_cookie=a8205a01a76b8294 in_seq=0 out_seq=0 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.947+0000 7f131b400640 10 client.15093 ms_handle_connect on v2:172.21.15.188:6838/57242391 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f131b400640 1 -- 172.21.15.16:0/1470243338 <== mds.2 v2:172.21.15.188:6838/57242391 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e6cc8aab40 con 0x55e6cc9be220 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f131b400640 10 client.15093 handle_command_reply: tid=0 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f133f248740 1 client.15093 shutdown 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f133f248740 1 -- 172.21.15.16:0/1470243338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a6fc0 msgr2=0x55e6cc9b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f133f248740 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b14d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1328009f50 tx=0x7f1328003dd0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f133f248740 1 -- 172.21.15.16:0/1470243338 shutdown_connections 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f133f248740 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x55e6cc9be220 0x55e6cc9bf690 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f133f248740 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9acb20 0x55e6cc9b8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f133f248740 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f133f248740 1 --2- 172.21.15.16:0/1470243338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b14d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f131b400640 0 client.15093 ms_handle_reset on v2:172.21.15.188:6838/57242391 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.948+0000 7f133f248740 1 -- 172.21.15.16:0/1470243338 >> 172.21.15.16:0/1470243338 conn(0x55e6cc9a2530 msgr2=0x55e6cc998f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.949+0000 7f133f248740 1 -- 172.21.15.16:0/1470243338 shutdown_connections 2024-04-13T03:33:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.949+0000 7f133f248740 1 -- 172.21.15.16:0/1470243338 wait complete. 2024-04-13T03:33:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.949+0000 7f133f248740 20 client.15093 trim_cache size 0 max 16384 2024-04-13T03:33:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133f248740 1 Processor -- start 2024-04-13T03:33:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133f248740 1 -- start start 2024-04-13T03:33:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133f248740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133f248740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133f248740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cc9a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133f248740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc9bd9d0 con 0x55e6cc9acb20 2024-04-13T03:33:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133f248740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc9be0a0 con 0x55e6cc9a79b0 2024-04-13T03:33:57.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133f248740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc9be770 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cc9a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f1339000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f1339000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:35486/0 (socket says 172.21.15.16:35486) 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f133a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cc9a8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59634/0 (socket says 172.21.15.16:59634) 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.969+0000 7f1339000640 1 -- 172.21.15.16:0/3298785343 learned_addr learned my addr 172.21.15.16:0/3298785343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.970+0000 7f1339a00640 1 --2- 172.21.15.16:0/3298785343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.970+0000 7f1339000640 1 -- 172.21.15.16:0/3298785343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 msgr2=0x55e6cc9aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.970+0000 7f1339000640 1 --2- 172.21.15.16:0/3298785343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.970+0000 7f1339000640 1 -- 172.21.15.16:0/3298785343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 msgr2=0x55e6cc9a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.970+0000 7f1339000640 1 --2- 172.21.15.16:0/3298785343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cc9a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.970+0000 7f1339000640 1 -- 172.21.15.16:0/3298785343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6cc9a32a0 con 0x55e6cc9a79b0 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.970+0000 7f133a400640 1 --2- 172.21.15.16:0/3298785343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cc9a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:57.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.970+0000 7f1339000640 1 --2- 172.21.15.16:0/3298785343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b8970 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f133419b8d0 tx=0x7f13200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b7dc1a94c751bf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.970+0000 7f1326000640 1 -- 172.21.15.16:0/3298785343 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320004460 con 0x55e6cc9a79b0 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f1326000640 1 -- 172.21.15.16:0/3298785343 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1320005500 con 0x55e6cc9a79b0 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f1326000640 1 -- 172.21.15.16:0/3298785343 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132002e290 con 0x55e6cc9a79b0 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f133f248740 1 -- 172.21.15.16:0/3298785343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 msgr2=0x55e6cc9b8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f133f248740 1 --2- 172.21.15.16:0/3298785343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b8970 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f133419b8d0 tx=0x7f13200046c0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f133f248740 1 -- 172.21.15.16:0/3298785343 shutdown_connections 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f133f248740 1 --2- 172.21.15.16:0/3298785343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cc9a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f133f248740 1 --2- 172.21.15.16:0/3298785343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b8970 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f133f248740 1 --2- 172.21.15.16:0/3298785343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f133f248740 1 -- 172.21.15.16:0/3298785343 >> 172.21.15.16:0/3298785343 conn(0x55e6cc9a2530 msgr2=0x55e6cc9a2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f133f248740 1 -- 172.21.15.16:0/3298785343 shutdown_connections 2024-04-13T03:33:57.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.971+0000 7f133f248740 1 -- 172.21.15.16:0/3298785343 wait complete. 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.972+0000 7f133f248740 1 -- start start 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.972+0000 7f133f248740 1 Processor -- start 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.972+0000 7f133f248740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.972+0000 7f133f248740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.972+0000 7f133f248740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cca221c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.972+0000 7f133f248740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc9bf510 con 0x55e6cc9acb20 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.972+0000 7f133f248740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc9bee40 con 0x55e6cc9a79b0 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.972+0000 7f133f248740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e6cc9bfbe0 con 0x55e6cc9a6fc0 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f133f248740 10 client.0 _ll_register_callbacks cb 0x7f1328004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f1339000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f1339000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:35494/0 (socket says 172.21.15.16:35494) 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f133a400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cca221c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f1339a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f1339000640 1 -- 172.21.15.16:0/4138945979 learned_addr learned my addr 172.21.15.16:0/4138945979 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f133a400640 1 -- 172.21.15.16:0/4138945979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 msgr2=0x55e6cc9b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f133a400640 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f133a400640 1 -- 172.21.15.16:0/4138945979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 msgr2=0x55e6cc9b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f133a400640 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f133a400640 1 -- 172.21.15.16:0/4138945979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6cc9b4410 con 0x55e6cc9acb20 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.973+0000 7f1339a00640 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.974+0000 7f133a400640 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cca221c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f132c006010 tx=0x7f132c032b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f902bea86404be server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.974+0000 7f131b400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.974+0000 7f131b400640 1 -- 172.21.15.16:0/4138945979 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c0d7070 con 0x55e6cc9acb20 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.974+0000 7f131b400640 1 -- 172.21.15.16:0/4138945979 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f132c0bee10 con 0x55e6cc9acb20 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.974+0000 7f133f248740 1 -- 172.21.15.16:0/4138945979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1320005500 con 0x55e6cc9acb20 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.974+0000 7f131b400640 1 -- 172.21.15.16:0/4138945979 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f132c0c2f60 con 0x55e6cc9acb20 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.974+0000 7f133f248740 10 client.15099 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:57.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.974+0000 7f133f248740 1 -- 172.21.15.16:0/4138945979 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e6cca23800 con 0x55e6cc9acb20 2024-04-13T03:33:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.974+0000 7f131b400640 1 -- 172.21.15.16:0/4138945979 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1320005500 con 0x55e6cc9acb20 2024-04-13T03:33:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.975+0000 7f131b400640 1 -- 172.21.15.16:0/4138945979 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f132c03c030 con 0x55e6cc9acb20 2024-04-13T03:33:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.975+0000 7f133f248740 10 client.15099 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.975+0000 7f133f248740 10 client.15099 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-13T03:33:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.975+0000 7f133f248740 20 client.15099 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.975+0000 7f133f248740 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x55e6cca284e0 0x55e6cca271f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.975+0000 7f133f248740 4 client.15099 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:33:57.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.975+0000 7f133f248740 1 -- 172.21.15.16:0/4138945979 --> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] -- command(tid 0: {"prefix": "status"}) -- 0x55e6cc9123b0 con 0x55e6cca284e0 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.975+0000 7f1339a00640 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x55e6cca284e0 0x55e6cca271f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f1339a00640 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x55e6cca284e0 0x55e6cca271f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=379ce6455d30dcef server_cookie=28e00576738de4fb in_seq=0 out_seq=0 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f131b400640 10 client.15099 ms_handle_connect on v2:172.21.15.188:6838/57242391 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f131b400640 1 -- 172.21.15.16:0/4138945979 <== mds.2 v2:172.21.15.188:6838/57242391 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e6cc9123b0 con 0x55e6cca284e0 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f131b400640 10 client.15099 handle_command_reply: tid=0 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 client.15099 shutdown 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 -- 172.21.15.16:0/4138945979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 msgr2=0x55e6cca221c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cca221c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f132c006010 tx=0x7f132c032b30 comp rx=0 tx=0).stop 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 -- 172.21.15.16:0/4138945979 shutdown_connections 2024-04-13T03:33:57.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e6cc9acb20 0x55e6cca221c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x55e6cca284e0 0x55e6cca271f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e6cc9a79b0 0x55e6cc9b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 --2- 172.21.15.16:0/4138945979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e6cc9a6fc0 0x55e6cc9b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f131b400640 0 client.15099 ms_handle_reset on v2:172.21.15.188:6838/57242391 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 -- 172.21.15.16:0/4138945979 >> 172.21.15.16:0/4138945979 conn(0x55e6cc9a2530 msgr2=0x55e6cc9a8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 -- 172.21.15.16:0/4138945979 shutdown_connections 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 1 -- 172.21.15.16:0/4138945979 wait complete. 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.976+0000 7f133f248740 20 client.15099 trim_cache size 0 max 16384 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:33:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24523, 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 112.940938117, 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 158.28495868300001 2024-04-13T03:33:57.983 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 -- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f131c0a3c30 msgr2=0x7f131c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f131c0a3c30 0x7f131c0a60f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55e6cc9a8210 tx=0x7f1320009770 comp rx=0 tx=0).stop 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 -- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 msgr2=0x7f13341a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 0x7f13341a1970 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f132c0325d0 tx=0x7f132c035d10 comp rx=0 tx=0).stop 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 -- 172.21.15.16:0/235463428 shutdown_connections 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f131c0a3c30 0x7f131c0a60f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1334103b30 0x7f13341a1970 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1334103170 0x7f133419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 --2- 172.21.15.16:0/235463428 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13341001a0 0x7f133419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 -- 172.21.15.16:0/235463428 >> 172.21.15.16:0/235463428 conn(0x7f13340fbaf0 msgr2=0x7f1334100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 -- 172.21.15.16:0/235463428 shutdown_connections 2024-04-13T03:33:57.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:57.977+0000 7f133ae00640 1 -- 172.21.15.16:0/235463428 wait complete. 2024-04-13T03:33:57.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-13T03:33:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.214+0000 7fe715800640 1 -- 172.21.15.16:0/2066919029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe710104100 msgr2=0x7fe710104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.214+0000 7fe715800640 1 --2- 172.21.15.16:0/2066919029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe710104100 0x7fe710104520 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe700009900 tx=0x7fe70002bc30 comp rx=0 tx=0).stop 2024-04-13T03:33:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.214+0000 7fe715800640 1 -- 172.21.15.16:0/2066919029 shutdown_connections 2024-04-13T03:33:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.214+0000 7fe715800640 1 --2- 172.21.15.16:0/2066919029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe710105cb0 0x7fe71010c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.214+0000 7fe715800640 1 --2- 172.21.15.16:0/2066919029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7101052f0 0x7fe710105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.214+0000 7fe715800640 1 --2- 172.21.15.16:0/2066919029 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe710104100 0x7fe710104520 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.214+0000 7fe715800640 1 -- 172.21.15.16:0/2066919029 >> 172.21.15.16:0/2066919029 conn(0x7fe7100ff670 msgr2=0x7fe710101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.215+0000 7fe715800640 1 -- 172.21.15.16:0/2066919029 shutdown_connections 2024-04-13T03:33:58.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.215+0000 7fe715800640 1 -- 172.21.15.16:0/2066919029 wait complete. 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.216+0000 7fe715800640 1 Processor -- start 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.216+0000 7fe715800640 1 -- start start 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.216+0000 7fe715800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe710104100 0x7fe71019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.216+0000 7fe715800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7101052f0 0x7fe71019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.216+0000 7fe715800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe710105cb0 0x7fe7101a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.216+0000 7fe715800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe71010fd60 con 0x7fe710105cb0 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.216+0000 7fe715800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe71010fbe0 con 0x7fe710104100 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.216+0000 7fe715800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe71010fee0 con 0x7fe7101052f0 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7101052f0 0x7fe71019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe714e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe710105cb0 0x7fe7101a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7101052f0 0x7fe71019a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53512/0 (socket says 172.21.15.16:53512) 2024-04-13T03:33:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe710104100 0x7fe71019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70f400640 1 -- 172.21.15.16:0/1174526857 learned_addr learned my addr 172.21.15.16:0/1174526857 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70f400640 1 -- 172.21.15.16:0/1174526857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe710104100 msgr2=0x7fe71019a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70f400640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe710104100 0x7fe71019a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70f400640 1 -- 172.21.15.16:0/1174526857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe710105cb0 msgr2=0x7fe7101a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70f400640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe710105cb0 0x7fe7101a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70f400640 1 -- 172.21.15.16:0/1174526857 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7101a2000 con 0x7fe7101052f0 2024-04-13T03:33:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.217+0000 7fe70f400640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7101052f0 0x7fe71019a920 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00e9f0 tx=0x7fe6fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.218+0000 7fe70cc00640 1 -- 172.21.15.16:0/1174526857 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc00cd80 con 0x7fe7101052f0 2024-04-13T03:33:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.218+0000 7fe70cc00640 1 -- 172.21.15.16:0/1174526857 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6fc010910 con 0x7fe7101052f0 2024-04-13T03:33:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.218+0000 7fe715800640 1 -- 172.21.15.16:0/1174526857 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7101a22f0 con 0x7fe7101052f0 2024-04-13T03:33:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.218+0000 7fe70cc00640 1 -- 172.21.15.16:0/1174526857 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc018990 con 0x7fe7101052f0 2024-04-13T03:33:58.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.218+0000 7fe715800640 1 -- 172.21.15.16:0/1174526857 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7101a2840 con 0x7fe7101052f0 2024-04-13T03:33:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.221+0000 7fe719c56740 1 Processor -- start 2024-04-13T03:33:58.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.221+0000 7fe719c56740 1 -- start start 2024-04-13T03:33:58.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.221+0000 7fe719c56740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 0x5573b8d77bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe719c56740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d7cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe719c56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d7f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe719c56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5573b8cdd990 con 0x5573b8d7d310 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe719c56740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5573b8c84530 con 0x5573b8d777b0 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe719c56740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5573b8cd9950 con 0x5573b8d781a0 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe70fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d7cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe70fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d7cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53516/0 (socket says 172.21.15.16:53516) 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe70fe00640 1 -- 172.21.15.16:0/858090992 learned_addr learned my addr 172.21.15.16:0/858090992 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe714e00640 1 --2- 172.21.15.16:0/858090992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 0x5573b8d77bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe70fe00640 1 -- 172.21.15.16:0/858090992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 msgr2=0x5573b8d77bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe70fe00640 1 --2- 172.21.15.16:0/858090992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 0x5573b8d77bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe70fe00640 1 -- 172.21.15.16:0/858090992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 msgr2=0x5573b8d7f7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-13T03:33:58.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe70fe00640 1 --2- 172.21.15.16:0/858090992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d7f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe70fe00640 1 -- 172.21.15.16:0/858090992 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573b89f02d0 con 0x5573b8d781a0 2024-04-13T03:33:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.222+0000 7fe70cc00640 1 -- 172.21.15.16:0/1174526857 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe6fc010ac0 con 0x7fe7101052f0 2024-04-13T03:33:58.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.223+0000 7fe70fe00640 1 --2- 172.21.15.16:0/858090992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d7cca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe700009900 tx=0x7fe7000095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbb93b31acd8d244 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.223+0000 7fe705600640 1 -- 172.21.15.16:0/858090992 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70002cb80 con 0x5573b8d781a0 2024-04-13T03:33:58.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.223+0000 7fe705600640 1 -- 172.21.15.16:0/858090992 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe700005040 con 0x5573b8d781a0 2024-04-13T03:33:58.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.223+0000 7fe705600640 1 -- 172.21.15.16:0/858090992 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7000048c0 con 0x5573b8d781a0 2024-04-13T03:33:58.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe719c56740 1 -- 172.21.15.16:0/858090992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 msgr2=0x5573b8d7cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe719c56740 1 --2- 172.21.15.16:0/858090992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d7cca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe700009900 tx=0x7fe7000095e0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe70cc00640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6f40a3bb0 0x7fe6f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe719c56740 1 -- 172.21.15.16:0/858090992 shutdown_connections 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe719c56740 1 --2- 172.21.15.16:0/858090992 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d7f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe719c56740 1 --2- 172.21.15.16:0/858090992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d7cca0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe719c56740 1 --2- 172.21.15.16:0/858090992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 0x5573b8d77bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe719c56740 1 -- 172.21.15.16:0/858090992 >> 172.21.15.16:0/858090992 conn(0x5573b8d72d20 msgr2=0x5573b8d73120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe70cc00640 1 -- 172.21.15.16:0/1174526857 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe6fc0d0b60 con 0x7fe7101052f0 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe719c56740 1 -- 172.21.15.16:0/858090992 shutdown_connections 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe714e00640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6f40a3bb0 0x7fe6f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.224+0000 7fe719c56740 1 -- 172.21.15.16:0/858090992 wait complete. 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe719c56740 1 -- start start 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe719c56740 1 Processor -- start 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe714e00640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6f40a3bb0 0x7fe6f40a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x5573b8d78a00 tx=0x7fe708009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe719c56740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d81d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe719c56740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d781a0 0x5573b8d82270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe719c56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d892b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe719c56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5573b8d80260 con 0x5573b8d7d310 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe719c56740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5573b8c978f0 con 0x5573b8d781a0 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe719c56740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5573b8d803e0 con 0x5573b8d777b0 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.225+0000 7fe719c56740 10 client.0 _ll_register_callbacks cb 0x5573b8c972e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d81d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe714e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d892b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d81d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53518/0 (socket says 172.21.15.16:53518) 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70fe00640 1 -- 172.21.15.16:0/2466891460 learned_addr learned my addr 172.21.15.16:0/2466891460 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70f400640 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d781a0 0x5573b8d82270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70fe00640 1 -- 172.21.15.16:0/2466891460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d781a0 msgr2=0x5573b8d82270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70fe00640 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d781a0 0x5573b8d82270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70fe00640 1 -- 172.21.15.16:0/2466891460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 msgr2=0x5573b8d892b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70fe00640 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d892b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70fe00640 1 -- 172.21.15.16:0/2466891460 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573b8aba6b0 con 0x5573b8d777b0 2024-04-13T03:33:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe70fe00640 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d81d30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe70002cfd0 tx=0x7fe70002ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b0519698b7c0b04 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.226+0000 7fe6fa000640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-13T03:33:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.227+0000 7fe6fa000640 1 -- 172.21.15.16:0/2466891460 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700004ad0 con 0x5573b8d777b0 2024-04-13T03:33:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.227+0000 7fe6fa000640 1 -- 172.21.15.16:0/2466891460 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe700004c70 con 0x5573b8d777b0 2024-04-13T03:33:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.227+0000 7fe719c56740 1 -- 172.21.15.16:0/2466891460 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe700005040 con 0x5573b8d777b0 2024-04-13T03:33:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.227+0000 7fe6fa000640 1 -- 172.21.15.16:0/2466891460 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe700005620 con 0x5573b8d777b0 2024-04-13T03:33:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.227+0000 7fe719c56740 10 client.25198 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.227+0000 7fe719c56740 1 -- 172.21.15.16:0/2466891460 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5573b8d802b0 con 0x5573b8d777b0 2024-04-13T03:33:58.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.227+0000 7fe6fa000640 1 -- 172.21.15.16:0/2466891460 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe700005040 con 0x5573b8d777b0 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.228+0000 7fe6fa000640 1 -- 172.21.15.16:0/2466891460 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fe7000454b0 con 0x5573b8d777b0 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.228+0000 7fe719c56740 10 client.25198 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.228+0000 7fe719c56740 10 client.25198 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.228+0000 7fe719c56740 20 client.25198 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.228+0000 7fe719c56740 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x5573b8d8e7c0 0x5573b8d8fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.228+0000 7fe719c56740 4 client.25198 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.228+0000 7fe719c56740 1 -- 172.21.15.16:0/2466891460 --> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573b8c83b40 con 0x5573b8d8e7c0 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.228+0000 7fe70f400640 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x5573b8d8e7c0 0x5573b8d8fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.229+0000 7fe70f400640 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x5573b8d8e7c0 0x5573b8d8fc30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b6d6dbaecc88d653 server_cookie=884af545412f8c4a in_seq=0 out_seq=0 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.229+0000 7fe6fa000640 10 client.25198 ms_handle_connect on v2:172.21.15.119:6836/477256695 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe6fa000640 1 -- 172.21.15.16:0/2466891460 <== mds.1 v2:172.21.15.119:6836/477256695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5573b8c83b40 con 0x5573b8d8e7c0 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe6fa000640 10 client.25198 handle_command_reply: tid=0 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 client.25198 shutdown 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 -- 172.21.15.16:0/2466891460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 msgr2=0x5573b8d81d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d81d30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe70002cfd0 tx=0x7fe70002ccb0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 -- 172.21.15.16:0/2466891460 shutdown_connections 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d892b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x5573b8d8e7c0 0x5573b8d8fc30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d781a0 0x5573b8d82270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 --2- 172.21.15.16:0/2466891460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d81d30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe6fa000640 0 client.25198 ms_handle_reset on v2:172.21.15.119:6836/477256695 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 -- 172.21.15.16:0/2466891460 >> 172.21.15.16:0/2466891460 conn(0x5573b8d72d20 msgr2=0x5573b8d71f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 -- 172.21.15.16:0/2466891460 shutdown_connections 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 1 -- 172.21.15.16:0/2466891460 wait complete. 2024-04-13T03:33:58.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.230+0000 7fe719c56740 20 client.25198 trim_cache size 0 max 16384 2024-04-13T03:33:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.250+0000 7fe719c56740 1 Processor -- start 2024-04-13T03:33:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe719c56740 1 -- start start 2024-04-13T03:33:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe719c56740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 0x5573b8d716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe719c56740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d89270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe719c56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d71bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe719c56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5573b8d8df70 con 0x5573b8d7d310 2024-04-13T03:33:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe719c56740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5573b8d8e640 con 0x5573b8d777b0 2024-04-13T03:33:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe719c56740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5573b8d8ed10 con 0x5573b8d781a0 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe714e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d71bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe70fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 0x5573b8d716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe70f400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d89270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe714e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d71bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59680/0 (socket says 172.21.15.16:59680) 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.251+0000 7fe714e00640 1 -- 172.21.15.16:0/2403884807 learned_addr learned my addr 172.21.15.16:0/2403884807 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.252+0000 7fe70fe00640 1 -- 172.21.15.16:0/2403884807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 msgr2=0x5573b8d89270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.252+0000 7fe70fe00640 1 --2- 172.21.15.16:0/2403884807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d89270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.252+0000 7fe70fe00640 1 -- 172.21.15.16:0/2403884807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 msgr2=0x5573b8d71bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.252+0000 7fe70fe00640 1 --2- 172.21.15.16:0/2403884807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d71bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.252+0000 7fe70fe00640 1 -- 172.21.15.16:0/2403884807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573b8d78580 con 0x5573b8d777b0 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.252+0000 7fe70f400640 1 --2- 172.21.15.16:0/2403884807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d89270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.252+0000 7fe70fe00640 1 --2- 172.21.15.16:0/2403884807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 0x5573b8d716b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe700045a40 tx=0x7fe700005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb36f1b01b92b048 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe704c00640 1 -- 172.21.15.16:0/2403884807 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7000057c0 con 0x5573b8d777b0 2024-04-13T03:33:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe704c00640 1 -- 172.21.15.16:0/2403884807 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe700045ac0 con 0x5573b8d777b0 2024-04-13T03:33:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe704c00640 1 -- 172.21.15.16:0/2403884807 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7000346c0 con 0x5573b8d777b0 2024-04-13T03:33:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe719c56740 1 -- 172.21.15.16:0/2403884807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 msgr2=0x5573b8d716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe719c56740 1 --2- 172.21.15.16:0/2403884807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 0x5573b8d716b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe700045a40 tx=0x7fe700005b50 comp rx=0 tx=0).stop 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe719c56740 1 -- 172.21.15.16:0/2403884807 shutdown_connections 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe719c56740 1 --2- 172.21.15.16:0/2403884807 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d7d310 0x5573b8d71bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe719c56740 1 --2- 172.21.15.16:0/2403884807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d781a0 0x5573b8d89270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe719c56740 1 --2- 172.21.15.16:0/2403884807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d777b0 0x5573b8d716b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe719c56740 1 -- 172.21.15.16:0/2403884807 >> 172.21.15.16:0/2403884807 conn(0x5573b8d72d20 msgr2=0x5573b8d73100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe719c56740 1 -- 172.21.15.16:0/2403884807 shutdown_connections 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.253+0000 7fe719c56740 1 -- 172.21.15.16:0/2403884807 wait complete. 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.254+0000 7fe719c56740 1 -- start start 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.254+0000 7fe719c56740 1 Processor -- start 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.254+0000 7fe719c56740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d87410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.254+0000 7fe719c56740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d781a0 0x5573b8d87950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.254+0000 7fe719c56740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d7d310 0x5573b8d87f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.254+0000 7fe719c56740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5573b8d8fab0 con 0x5573b8d781a0 2024-04-13T03:33:58.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.254+0000 7fe719c56740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5573b8d8f3e0 con 0x5573b8d7d310 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.254+0000 7fe719c56740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5573b8d90180 con 0x5573b8d777b0 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.254+0000 7fe719c56740 10 client.0 _ll_register_callbacks cb 0x7fe70002c330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d781a0 0x5573b8d87950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70f400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d781a0 0x5573b8d87950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59688/0 (socket says 172.21.15.16:59688) 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70f400640 1 -- 172.21.15.16:0/3343475377 learned_addr learned my addr 172.21.15.16:0/3343475377 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe714e00640 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d7d310 0x5573b8d87f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70fe00640 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d87410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70f400640 1 -- 172.21.15.16:0/3343475377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 msgr2=0x5573b8d87410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70f400640 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d87410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70f400640 1 -- 172.21.15.16:0/3343475377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d7d310 msgr2=0x5573b8d87f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70f400640 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d7d310 0x5573b8d87f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70f400640 1 -- 172.21.15.16:0/3343475377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573b8d887b0 con 0x5573b8d781a0 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe70f400640 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d781a0 0x5573b8d87950 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00cfd0 tx=0x7fe6fc018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826fcf54eb6c7d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.255+0000 7fe6fa000640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:33:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.256+0000 7fe6fa000640 1 -- 172.21.15.16:0/3343475377 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc099130 con 0x5573b8d781a0 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.256+0000 7fe6fa000640 1 -- 172.21.15.16:0/3343475377 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe6fc0a51f0 con 0x5573b8d781a0 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.256+0000 7fe719c56740 1 -- 172.21.15.16:0/3343475377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe700045ac0 con 0x5573b8d781a0 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.256+0000 7fe6fa000640 1 -- 172.21.15.16:0/3343475377 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe6fc0992d0 con 0x5573b8d781a0 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.256+0000 7fe719c56740 10 client.15114 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.256+0000 7fe719c56740 1 -- 172.21.15.16:0/3343475377 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5573b8df4ac0 con 0x5573b8d781a0 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.256+0000 7fe6fa000640 1 -- 172.21.15.16:0/3343475377 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe700045ac0 con 0x5573b8d781a0 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.256+0000 7fe6fa000640 1 -- 172.21.15.16:0/3343475377 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fe6fc09bfa0 con 0x5573b8d781a0 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.256+0000 7fe719c56740 10 client.15114 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.257+0000 7fe719c56740 10 client.15114 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.257+0000 7fe719c56740 20 client.15114 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.257+0000 7fe719c56740 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x5573b8df96e0 0x5573b8df5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.257+0000 7fe719c56740 4 client.15114 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:33:58.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.257+0000 7fe719c56740 1 -- 172.21.15.16:0/3343475377 --> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] -- command(tid 0: {"prefix": "status"}) -- 0x5573b8ceb3b0 con 0x5573b8df96e0 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.257+0000 7fe70fe00640 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x5573b8df96e0 0x5573b8df5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.257+0000 7fe70fe00640 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x5573b8df96e0 0x5573b8df5b20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c48302425809c723 server_cookie=d225eb9ba68a157f in_seq=0 out_seq=0 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.257+0000 7fe6fa000640 10 client.15114 ms_handle_connect on v2:172.21.15.119:6836/477256695 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe6fa000640 1 -- 172.21.15.16:0/3343475377 <== mds.1 v2:172.21.15.119:6836/477256695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5573b8ceb3b0 con 0x5573b8df96e0 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe6fa000640 10 client.15114 handle_command_reply: tid=0 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 client.15114 shutdown 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 -- 172.21.15.16:0/3343475377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d781a0 msgr2=0x5573b8d87950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d781a0 0x5573b8d87950 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00cfd0 tx=0x7fe6fc018420 comp rx=0 tx=0).stop 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 -- 172.21.15.16:0/3343475377 shutdown_connections 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x5573b8df96e0 0x5573b8df5b20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573b8d7d310 0x5573b8d87f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5573b8d781a0 0x5573b8d87950 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 --2- 172.21.15.16:0/3343475377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5573b8d777b0 0x5573b8d87410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe6fa000640 0 client.15114 ms_handle_reset on v2:172.21.15.119:6836/477256695 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 -- 172.21.15.16:0/3343475377 >> 172.21.15.16:0/3343475377 conn(0x5573b8d72d20 msgr2=0x5573b8d75f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 -- 172.21.15.16:0/3343475377 shutdown_connections 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 1 -- 172.21.15.16:0/3343475377 wait complete. 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.258+0000 7fe719c56740 20 client.15114 trim_cache size 0 max 16384 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24530, 2024-04-13T03:33:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:33:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:33:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:33:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 114.233029984, 2024-04-13T03:33:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:33:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:33:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:33:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 161.529588399 2024-04-13T03:33:58.264 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:33:58.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 -- 172.21.15.16:0/1174526857 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6f40a3bb0 msgr2=0x7fe6f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6f40a3bb0 0x7fe6f40a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x5573b8d78a00 tx=0x7fe708009600 comp rx=0 tx=0).stop 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 -- 172.21.15.16:0/1174526857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7101052f0 msgr2=0x7fe71019a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7101052f0 0x7fe71019a920 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00e9f0 tx=0x7fe6fc00eeb0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 -- 172.21.15.16:0/1174526857 shutdown_connections 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6f40a3bb0 0x7fe6f40a6070 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe710105cb0 0x7fe7101a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe7101052f0 0x7fe71019a920 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 --2- 172.21.15.16:0/1174526857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe710104100 0x7fe71019a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 -- 172.21.15.16:0/1174526857 >> 172.21.15.16:0/1174526857 conn(0x7fe7100ff670 msgr2=0x7fe7101085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 -- 172.21.15.16:0/1174526857 shutdown_connections 2024-04-13T03:33:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.259+0000 7fe715800640 1 -- 172.21.15.16:0/1174526857 wait complete. 2024-04-13T03:33:58.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-13T03:33:58.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.487+0000 7f4b76200640 1 -- 172.21.15.16:0/4102963820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b70074a20 msgr2=0x7f4b70074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.487+0000 7f4b76200640 1 --2- 172.21.15.16:0/4102963820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b70074a20 0x7f4b70074ea0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009930 tx=0x7f4b6002bc20 comp rx=0 tx=0).stop 2024-04-13T03:33:58.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.487+0000 7f4b76200640 1 -- 172.21.15.16:0/4102963820 shutdown_connections 2024-04-13T03:33:58.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.487+0000 7f4b76200640 1 --2- 172.21.15.16:0/4102963820 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b7010e4a0 0x7f4b70110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.487+0000 7f4b76200640 1 --2- 172.21.15.16:0/4102963820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b70074a20 0x7f4b70074ea0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.487+0000 7f4b76200640 1 --2- 172.21.15.16:0/4102963820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b70076000 0x7f4b70074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.487+0000 7f4b76200640 1 -- 172.21.15.16:0/4102963820 >> 172.21.15.16:0/4102963820 conn(0x7f4b700fd460 msgr2=0x7f4b700ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.487+0000 7f4b76200640 1 -- 172.21.15.16:0/4102963820 shutdown_connections 2024-04-13T03:33:58.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.487+0000 7f4b76200640 1 -- 172.21.15.16:0/4102963820 wait complete. 2024-04-13T03:33:58.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.488+0000 7f4b76200640 1 Processor -- start 2024-04-13T03:33:58.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.488+0000 7f4b76200640 1 -- start start 2024-04-13T03:33:58.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.488+0000 7f4b76200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074a20 0x7f4b7019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.488+0000 7f4b76200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b70076000 0x7f4b7019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.488+0000 7f4b76200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7010e4a0 0x7f4b701a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.488+0000 7f4b76200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701141b0 con 0x7f4b70074a20 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.488+0000 7f4b76200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70114030 con 0x7f4b7010e4a0 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.488+0000 7f4b76200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70114330 con 0x7f4b70076000 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b74e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074a20 0x7f4b7019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b70076000 0x7f4b7019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b74e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074a20 0x7f4b7019e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59698/0 (socket says 172.21.15.16:59698) 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b74e00640 1 -- 172.21.15.16:0/1882112780 learned_addr learned my addr 172.21.15.16:0/1882112780 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b75800640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7010e4a0 0x7f4b701a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b74e00640 1 -- 172.21.15.16:0/1882112780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b70076000 msgr2=0x7f4b7019ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b74e00640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b70076000 0x7f4b7019ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b74e00640 1 -- 172.21.15.16:0/1882112780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7010e4a0 msgr2=0x7f4b701a5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b74e00640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7010e4a0 0x7f4b701a5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b74e00640 1 -- 172.21.15.16:0/1882112780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b701a6400 con 0x7f4b70074a20 2024-04-13T03:33:58.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b70076000 0x7f4b7019ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:58.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.489+0000 7f4b74e00640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074a20 0x7f4b7019e7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400d950 tx=0x7f4b6400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.490+0000 7f4b6d600640 1 -- 172.21.15.16:0/1882112780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64017030 con 0x7f4b70074a20 2024-04-13T03:33:58.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.490+0000 7f4b6d600640 1 -- 172.21.15.16:0/1882112780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b64004020 con 0x7f4b70074a20 2024-04-13T03:33:58.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.490+0000 7f4b76200640 1 -- 172.21.15.16:0/1882112780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b701a66f0 con 0x7f4b70074a20 2024-04-13T03:33:58.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.490+0000 7f4b6d600640 1 -- 172.21.15.16:0/1882112780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b640047a0 con 0x7f4b70074a20 2024-04-13T03:33:58.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.490+0000 7f4b76200640 1 -- 172.21.15.16:0/1882112780 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b701a6c10 con 0x7f4b70074a20 2024-04-13T03:33:58.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.494+0000 7f4b7a7a1740 1 Processor -- start 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.494+0000 7f4b7a7a1740 1 -- start start 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b6d600640 1 -- 172.21.15.16:0/1882112780 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4b640041d0 con 0x7f4b70074a20 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afdd4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afdd9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdda310 0x5570afddc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5570afd3a990 con 0x5570afdd51a0 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5570afce1530 con 0x5570afdda310 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5570afd36950 con 0x5570afdd47b0 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b74e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afdd9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b75800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afdd4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b75800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afdd4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53552/0 (socket says 172.21.15.16:53552) 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b74e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afdd9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59700/0 (socket says 172.21.15.16:59700) 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b75800640 1 -- 172.21.15.16:0/2646638000 learned_addr learned my addr 172.21.15.16:0/2646638000 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.495+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/2646638000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdda310 0x5570afddc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.496+0000 7f4b75800640 1 -- 172.21.15.16:0/2646638000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdda310 msgr2=0x5570afddc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.496+0000 7f4b75800640 1 --2- 172.21.15.16:0/2646638000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdda310 0x5570afddc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.496+0000 7f4b75800640 1 -- 172.21.15.16:0/2646638000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 msgr2=0x5570afdd9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.496+0000 7f4b75800640 1 --2- 172.21.15.16:0/2646638000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afdd9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.496+0000 7f4b75800640 1 -- 172.21.15.16:0/2646638000 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570afa4d2d0 con 0x5570afdd47b0 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.496+0000 7f4b75800640 1 --2- 172.21.15.16:0/2646638000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afdd4bd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009f50 tx=0x7f4b680118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bf1b94be5b22458 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.496+0000 7f4b5e000640 1 -- 172.21.15.16:0/2646638000 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68004d20 con 0x5570afdd47b0 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.496+0000 7f4b5e000640 1 -- 172.21.15.16:0/2646638000 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b68004050 con 0x5570afdd47b0 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.496+0000 7f4b5e000640 1 -- 172.21.15.16:0/2646638000 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68005280 con 0x5570afdd47b0 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b6d600640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4b580a3bb0 0x7f4b580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b6d600640 1 -- 172.21.15.16:0/1882112780 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4b640d01c0 con 0x7f4b70074a20 2024-04-13T03:33:58.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b74e00640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4b580a3bb0 0x7f4b580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2646638000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 msgr2=0x5570afdd4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2646638000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afdd4bd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009f50 tx=0x7f4b680118d0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2646638000 shutdown_connections 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2646638000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdda310 0x5570afddc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2646638000 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afdd9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2646638000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afdd4bd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.497+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2646638000 >> 172.21.15.16:0/2646638000 conn(0x5570afdcfd20 msgr2=0x5570afdd0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.498+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2646638000 shutdown_connections 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.498+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2646638000 wait complete. 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.498+0000 7f4b74e00640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4b580a3bb0 0x7f4b580a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4b640cdc60 tx=0x7f4b640d3ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.498+0000 7f4b7a7a1740 1 -- start start 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.498+0000 7f4b7a7a1740 1 Processor -- start 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 0x5570afddecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afddf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 0x5570afde6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5570afddd2c0 con 0x5570afdd51a0 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5570afcf48f0 con 0x5570afdd47b0 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5570afddd440 con 0x5570afdda310 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b7a7a1740 10 client.0 _ll_register_callbacks cb 0x5570afcf42e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afddf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afddf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59712/0 (socket says 172.21.15.16:59712) 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b6fe00640 1 -- 172.21.15.16:0/2507327023 learned_addr learned my addr 172.21.15.16:0/2507327023 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b75800640 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 0x5570afde6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b75800640 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 0x5570afddecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b6fe00640 1 -- 172.21.15.16:0/2507327023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 msgr2=0x5570afde6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 0x5570afde6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b6fe00640 1 -- 172.21.15.16:0/2507327023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 msgr2=0x5570afddecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 0x5570afddecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b6fe00640 1 -- 172.21.15.16:0/2507327023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570afb176b0 con 0x5570afdd51a0 2024-04-13T03:33:58.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.499+0000 7f4b75800640 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 0x5570afde6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.500+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afddf230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005ab0 tx=0x7f4b600048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7093606df4f3e584 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.500+0000 7f4b52a00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.500+0000 7f4b52a00640 1 -- 172.21.15.16:0/2507327023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60005d50 con 0x5570afdd51a0 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.500+0000 7f4b52a00640 1 -- 172.21.15.16:0/2507327023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4b60004c50 con 0x5570afdd51a0 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.500+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2507327023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b68004050 con 0x5570afdd51a0 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.500+0000 7f4b52a00640 1 -- 172.21.15.16:0/2507327023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4b6002cd20 con 0x5570afdd51a0 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.500+0000 7f4b7a7a1740 10 client.15126 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.500+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2507327023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5570afde9650 con 0x5570afdd51a0 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.501+0000 7f4b52a00640 1 -- 172.21.15.16:0/2507327023 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b68004050 con 0x5570afdd51a0 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.501+0000 7f4b52a00640 1 -- 172.21.15.16:0/2507327023 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f4b60003970 con 0x5570afdd51a0 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.501+0000 7f4b7a7a1740 10 client.15126 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.501+0000 7f4b7a7a1740 10 client.15126 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-13T03:33:58.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.501+0000 7f4b7a7a1740 20 client.15126 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.501+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x5570afdebcc0 0x5570afded130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.501+0000 7f4b7a7a1740 4 client.15126 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:33:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.501+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2507327023 --> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570afce0b40 con 0x5570afdebcc0 2024-04-13T03:33:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.502+0000 7f4b75800640 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x5570afdebcc0 0x5570afded130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.502+0000 7f4b75800640 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x5570afdebcc0 0x5570afded130 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=413987c44b1bd428 server_cookie=3fc3c8ec9e6a2037 in_seq=0 out_seq=0 2024-04-13T03:33:58.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.502+0000 7f4b52a00640 10 client.15126 ms_handle_connect on v2:172.21.15.119:6838/3722020849 2024-04-13T03:33:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b52a00640 1 -- 172.21.15.16:0/2507327023 <== mds.0 v2:172.21.15.119:6838/3722020849 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570afce0b40 con 0x5570afdebcc0 2024-04-13T03:33:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b52a00640 10 client.15126 handle_command_reply: tid=0 2024-04-13T03:33:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b7a7a1740 1 client.15126 shutdown 2024-04-13T03:33:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2507327023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 msgr2=0x5570afddf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afddf230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005ab0 tx=0x7f4b600048d0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2507327023 shutdown_connections 2024-04-13T03:33:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x5570afdebcc0 0x5570afded130 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 0x5570afde6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afddf230 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b52a00640 0 client.15126 ms_handle_reset on v2:172.21.15.119:6838/3722020849 2024-04-13T03:33:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.503+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2507327023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 0x5570afddecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.504+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2507327023 >> 172.21.15.16:0/2507327023 conn(0x5570afdcfd20 msgr2=0x5570afdcef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.504+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2507327023 shutdown_connections 2024-04-13T03:33:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.504+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2507327023 wait complete. 2024-04-13T03:33:58.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.504+0000 7f4b7a7a1740 20 client.15126 trim_cache size 0 max 16384 2024-04-13T03:33:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.524+0000 7f4b7a7a1740 1 Processor -- start 2024-04-13T03:33:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.524+0000 7f4b7a7a1740 1 -- start start 2024-04-13T03:33:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.524+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 0x5570afdd66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.524+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afde6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.524+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 0x5570afdd6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.524+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5570afdeb470 con 0x5570afdd51a0 2024-04-13T03:33:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.524+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5570afdebb40 con 0x5570afdd47b0 2024-04-13T03:33:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.524+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5570afdec210 con 0x5570afdda310 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.524+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afde6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b75800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 0x5570afdd66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afde6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59720/0 (socket says 172.21.15.16:59720) 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b6fe00640 1 -- 172.21.15.16:0/3855619038 learned_addr learned my addr 172.21.15.16:0/3855619038 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b75800640 1 --2- 172.21.15.16:0/3855619038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 0x5570afdd6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b6fe00640 1 -- 172.21.15.16:0/3855619038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 msgr2=0x5570afdd6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/3855619038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 0x5570afdd6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b6fe00640 1 -- 172.21.15.16:0/3855619038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 msgr2=0x5570afdd66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/3855619038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 0x5570afdd66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b6fe00640 1 -- 172.21.15.16:0/3855619038 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570afdd5580 con 0x5570afdd51a0 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/3855619038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afde6230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4b60003d90 tx=0x7f4b6002bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8c694d1a7a7e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.525+0000 7f4b5d600640 1 -- 172.21.15.16:0/3855619038 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60000e60 con 0x5570afdd51a0 2024-04-13T03:33:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b5d600640 1 -- 172.21.15.16:0/3855619038 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6002cd20 con 0x5570afdd51a0 2024-04-13T03:33:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b5d600640 1 -- 172.21.15.16:0/3855619038 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60034c20 con 0x5570afdd51a0 2024-04-13T03:33:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/3855619038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 msgr2=0x5570afde6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/3855619038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afde6230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4b60003d90 tx=0x7f4b6002bc20 comp rx=0 tx=0).stop 2024-04-13T03:33:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/3855619038 shutdown_connections 2024-04-13T03:33:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/3855619038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdda310 0x5570afdd6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/3855619038 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdd51a0 0x5570afde6230 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/3855619038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd47b0 0x5570afdd66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/3855619038 >> 172.21.15.16:0/3855619038 conn(0x5570afdcfd20 msgr2=0x5570afddd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/3855619038 shutdown_connections 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.526+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/3855619038 wait complete. 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.527+0000 7f4b7a7a1740 1 -- start start 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.527+0000 7f4b7a7a1740 1 Processor -- start 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.527+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afde1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.527+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd51a0 0x5570afde1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.527+0000 7f4b7a7a1740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdda310 0x5570afde4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.527+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5570afdecfb0 con 0x5570afdda310 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.527+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5570afdec8e0 con 0x5570afdd51a0 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.527+0000 7f4b7a7a1740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x5570afded680 con 0x5570afdd47b0 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.527+0000 7f4b7a7a1740 10 client.0 _ll_register_callbacks cb 0x7f4b60004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b75800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afde1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b75800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdda310 0x5570afde4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd51a0 0x5570afde1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd51a0 0x5570afde1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:35604/0 (socket says 172.21.15.16:35604) 2024-04-13T03:33:58.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b75800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afde1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53578/0 (socket says 172.21.15.16:53578) 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b6fe00640 1 -- 172.21.15.16:0/2486003341 learned_addr learned my addr 172.21.15.16:0/2486003341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b6fe00640 1 -- 172.21.15.16:0/2486003341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 msgr2=0x5570afde1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afde1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b6fe00640 1 -- 172.21.15.16:0/2486003341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdda310 msgr2=0x5570afde4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdda310 0x5570afde4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b6fe00640 1 -- 172.21.15.16:0/2486003341 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570afde5590 con 0x5570afdd51a0 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b6fe00640 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd51a0 0x5570afde1ff0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009850 tx=0x7f4b60034f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaf5b989a8fcc164 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.528+0000 7f4b52a00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.529+0000 7f4b52a00640 1 -- 172.21.15.16:0/2486003341 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6005c070 con 0x5570afdd51a0 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.529+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2486003341 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b6002cd20 con 0x5570afdd51a0 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.529+0000 7f4b52a00640 1 -- 172.21.15.16:0/2486003341 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4b60003800 con 0x5570afdd51a0 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.529+0000 7f4b52a00640 1 -- 172.21.15.16:0/2486003341 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4b6002c2a0 con 0x5570afdd51a0 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.529+0000 7f4b7a7a1740 10 client.25238 fetch_fsmap learned FSMap version 26 2024-04-13T03:33:58.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.529+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2486003341 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5570afe51cd0 con 0x5570afdd51a0 2024-04-13T03:33:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.529+0000 7f4b52a00640 1 -- 172.21.15.16:0/2486003341 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6002cd20 con 0x5570afdd51a0 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.530+0000 7f4b52a00640 1 -- 172.21.15.16:0/2486003341 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f4b6004b480 con 0x5570afdd51a0 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.530+0000 7f4b7a7a1740 10 client.25238 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.530+0000 7f4b7a7a1740 10 client.25238 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.530+0000 7f4b7a7a1740 20 client.25238 populate_metadata read hostname 'smithi016' 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.530+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x5570afe56590 0x5570afe552a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.530+0000 7f4b7a7a1740 4 client.25238 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.530+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2486003341 --> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] -- command(tid 0: {"prefix": "status"}) -- 0x5570afd483b0 con 0x5570afe56590 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.530+0000 7f4b75800640 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x5570afe56590 0x5570afe552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b75800640 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x5570afe56590 0x5570afe552a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fd07d6ec275678e server_cookie=2a4590745107cc30 in_seq=0 out_seq=0 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b52a00640 10 client.25238 ms_handle_connect on v2:172.21.15.119:6838/3722020849 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b52a00640 1 -- 172.21.15.16:0/2486003341 <== mds.0 v2:172.21.15.119:6838/3722020849 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5570afd483b0 con 0x5570afe56590 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b52a00640 10 client.25238 handle_command_reply: tid=0 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 client.25238 shutdown 2024-04-13T03:33:58.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2486003341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd51a0 msgr2=0x5570afde1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd51a0 0x5570afde1ff0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009850 tx=0x7f4b60034f40 comp rx=0 tx=0).stop 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2486003341 shutdown_connections 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x5570afe56590 0x5570afe552a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x5570afdda310 0x5570afde4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5570afdd51a0 0x5570afde1ff0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 --2- 172.21.15.16:0/2486003341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5570afdd47b0 0x5570afde1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2486003341 >> 172.21.15.16:0/2486003341 conn(0x5570afdcfd20 msgr2=0x5570afdd66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b52a00640 0 client.25238 ms_handle_reset on v2:172.21.15.119:6838/3722020849 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2486003341 shutdown_connections 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.531+0000 7f4b7a7a1740 1 -- 172.21.15.16:0/2486003341 wait complete. 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b7a7a1740 20 client.25238 trim_cache size 0 max 16384 2024-04-13T03:33:58.536 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24587, 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 140.979837742, 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 147.35558055499999 2024-04-13T03:33:58.537 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:33:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 -- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4b580a3bb0 msgr2=0x7f4b580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4b580a3bb0 0x7f4b580a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4b640cdc60 tx=0x7f4b640d3ca0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 -- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074a20 msgr2=0x7f4b7019e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:33:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074a20 0x7f4b7019e7e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400d950 tx=0x7f4b6400de10 comp rx=0 tx=0).stop 2024-04-13T03:33:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 -- 172.21.15.16:0/1882112780 shutdown_connections 2024-04-13T03:33:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4b580a3bb0 0x7f4b580a6070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7010e4a0 0x7f4b701a5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b70076000 0x7f4b7019ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 --2- 172.21.15.16:0/1882112780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b70074a20 0x7f4b7019e7e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:33:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 -- 172.21.15.16:0/1882112780 >> 172.21.15.16:0/1882112780 conn(0x7f4b700fd460 msgr2=0x7f4b70101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:33:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 -- 172.21.15.16:0/1882112780 shutdown_connections 2024-04-13T03:33:58.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:33:58.532+0000 7f4b76200640 1 -- 172.21.15.16:0/1882112780 wait complete. 2024-04-13T03:33:58.551 INFO:tasks.cephadm:Setup complete, yielding 2024-04-13T03:33:58.551 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-13T03:33:58.564 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T03:33:58.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch status' 2024-04-13T03:33:59.020 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:33:59.522 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:33:59 smithi016 ceph-mon[31772]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-13T03:33:59.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:33:59 smithi119 ceph-mon[35835]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-13T03:33:59.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:33:59 smithi188 ceph-mon[36398]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.330+0000 7fd89f800700 1 -- 172.21.15.16:0/1138826641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd898076a80 msgr2=0x7fd898074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.330+0000 7fd89f800700 1 --2- 172.21.15.16:0/1138826641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd898076a80 0x7fd898074ee0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd8880098f0 tx=0x7fd888009c00 comp rx=0 tx=0).stop 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.330+0000 7fd89f800700 1 -- 172.21.15.16:0/1138826641 shutdown_connections 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.330+0000 7fd89f800700 1 --2- 172.21.15.16:0/1138826641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd89810aa40 0x7fd89810ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.330+0000 7fd89f800700 1 --2- 172.21.15.16:0/1138826641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd898076a80 0x7fd898074ee0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.330+0000 7fd89f800700 1 --2- 172.21.15.16:0/1138826641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8980754b0 0x7fd898075930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.330+0000 7fd89f800700 1 -- 172.21.15.16:0/1138826641 >> 172.21.15.16:0/1138826641 conn(0x7fd8980fe100 msgr2=0x7fd898100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.330+0000 7fd89f800700 1 -- 172.21.15.16:0/1138826641 shutdown_connections 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.330+0000 7fd89f800700 1 -- 172.21.15.16:0/1138826641 wait complete. 2024-04-13T03:34:01.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.332+0000 7fd89f800700 1 Processor -- start 2024-04-13T03:34:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.332+0000 7fd89f800700 1 -- start start 2024-04-13T03:34:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.332+0000 7fd89f800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8980754b0 0x7fd89819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.332+0000 7fd89f800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89810aa40 0x7fd89819b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.332+0000 7fd89f800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8981a0720 0x7fd8981a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.332+0000 7fd89f800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd898110920 con 0x7fd89810aa40 2024-04-13T03:34:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.332+0000 7fd89f800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd8981107a0 con 0x7fd8981a0720 2024-04-13T03:34:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.332+0000 7fd89f800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd898110aa0 con 0x7fd8980754b0 2024-04-13T03:34:01.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd88fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89810aa40 0x7fd89819b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd89ce00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8980754b0 0x7fd89819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd88fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89810aa40 0x7fd89819b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39960/0 (socket says 172.21.15.16:39960) 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd89ce00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8980754b0 0x7fd89819b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:59174/0 (socket says 172.21.15.16:59174) 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd88fe00700 1 -- 172.21.15.16:0/374141590 learned_addr learned my addr 172.21.15.16:0/374141590 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd89d800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8981a0720 0x7fd8981a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd89ce00700 1 -- 172.21.15.16:0/374141590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8981a0720 msgr2=0x7fd8981a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd89ce00700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8981a0720 0x7fd8981a2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd89ce00700 1 -- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89810aa40 msgr2=0x7fd89819b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd89ce00700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89810aa40 0x7fd89819b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd89ce00700 1 -- 172.21.15.16:0/374141590 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8981a31b0 con 0x7fd8980754b0 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.333+0000 7fd88fe00700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89810aa40 0x7fd89819b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.334+0000 7fd89ce00700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8980754b0 0x7fd89819b1c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd888008000 tx=0x7fd888005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:01.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.334+0000 7fd896000700 1 -- 172.21.15.16:0/374141590 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888005650 con 0x7fd8980754b0 2024-04-13T03:34:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.334+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8981a3440 con 0x7fd8980754b0 2024-04-13T03:34:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.334+0000 7fd896000700 1 -- 172.21.15.16:0/374141590 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd88800ba40 con 0x7fd8980754b0 2024-04-13T03:34:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.334+0000 7fd896000700 1 -- 172.21.15.16:0/374141590 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888003c60 con 0x7fd8980754b0 2024-04-13T03:34:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.334+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8981a3890 con 0x7fd8980754b0 2024-04-13T03:34:01.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.335+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8980697a0 con 0x7fd8980754b0 2024-04-13T03:34:01.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.337+0000 7fd896000700 1 -- 172.21.15.16:0/374141590 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd88800b560 con 0x7fd8980754b0 2024-04-13T03:34:01.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.339+0000 7fd896000700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd87c0a3c40 0x7fd87c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:01.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.339+0000 7fd896000700 1 -- 172.21.15.16:0/374141590 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd8880d0ff0 con 0x7fd8980754b0 2024-04-13T03:34:01.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.339+0000 7fd88fe00700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd87c0a3c40 0x7fd87c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:01.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.340+0000 7fd88fe00700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd87c0a3c40 0x7fd87c0a6100 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd880005950 tx=0x7fd880009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.341+0000 7fd896000700 1 -- 172.21.15.16:0/374141590 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd888013030 con 0x7fd8980754b0 2024-04-13T03:34:01.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:01 smithi119 ceph-mon[35835]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-13T03:34:01.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.600+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fd898062ef0 con 0x7fd87c0a3c40 2024-04-13T03:34:01.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:01 smithi016 ceph-mon[31772]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-13T03:34:01.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.602+0000 7fd896000700 1 -- 172.21.15.16:0/374141590 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fd898062ef0 con 0x7fd87c0a3c40 2024-04-13T03:34:01.604 INFO:teuthology.orchestra.run.smithi016.stdout:Backend: cephadm 2024-04-13T03:34:01.604 INFO:teuthology.orchestra.run.smithi016.stdout:Available: Yes 2024-04-13T03:34:01.604 INFO:teuthology.orchestra.run.smithi016.stdout:Paused: No 2024-04-13T03:34:01.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.606+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd87c0a3c40 msgr2=0x7fd87c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:01.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.606+0000 7fd89f800700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd87c0a3c40 0x7fd87c0a6100 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd880005950 tx=0x7fd880009310 comp rx=0 tx=0).stop 2024-04-13T03:34:01.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.606+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8980754b0 msgr2=0x7fd89819b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:01.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.606+0000 7fd89f800700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8980754b0 0x7fd89819b1c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd888008000 tx=0x7fd888005250 comp rx=0 tx=0).stop 2024-04-13T03:34:01.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.607+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 shutdown_connections 2024-04-13T03:34:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.607+0000 7fd89f800700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd87c0a3c40 0x7fd87c0a6100 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.607+0000 7fd89f800700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8981a0720 0x7fd8981a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.607+0000 7fd89f800700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8980754b0 0x7fd89819b1c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.607+0000 7fd89f800700 1 --2- 172.21.15.16:0/374141590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89810aa40 0x7fd89819b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.607+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 >> 172.21.15.16:0/374141590 conn(0x7fd8980fe100 msgr2=0x7fd898106990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.607+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 shutdown_connections 2024-04-13T03:34:01.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:01.607+0000 7fd89f800700 1 -- 172.21.15.16:0/374141590 wait complete. 2024-04-13T03:34:01.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:01 smithi188 ceph-mon[36398]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-13T03:34:02.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-13T03:34:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:34:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:03 smithi016 ceph-mon[31772]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:03 smithi016 ceph-mon[31772]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-13T03:34:03.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:03 smithi119 ceph-mon[35835]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:03.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:03 smithi119 ceph-mon[35835]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-13T03:34:03.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:03 smithi188 ceph-mon[36398]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:03.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:03 smithi188 ceph-mon[36398]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-13T03:34:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.156+0000 7ff0a0600700 1 -- 172.21.15.16:0/2094918677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff098076ae0 msgr2=0x7ff098074f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.156+0000 7ff0a0600700 1 --2- 172.21.15.16:0/2094918677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff098076ae0 0x7ff098074f40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009990 tx=0x7ff08c009ca0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.156+0000 7ff0a0600700 1 -- 172.21.15.16:0/2094918677 shutdown_connections 2024-04-13T03:34:05.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.156+0000 7ff0a0600700 1 --2- 172.21.15.16:0/2094918677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff09810f0f0 0x7ff0981114e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.156+0000 7ff0a0600700 1 --2- 172.21.15.16:0/2094918677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff098076ae0 0x7ff098074f40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.156+0000 7ff0a0600700 1 --2- 172.21.15.16:0/2094918677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff098075510 0x7ff098075990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.156+0000 7ff0a0600700 1 -- 172.21.15.16:0/2094918677 >> 172.21.15.16:0/2094918677 conn(0x7ff0980fe0b0 msgr2=0x7ff0981004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:05.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.156+0000 7ff0a0600700 1 -- 172.21.15.16:0/2094918677 shutdown_connections 2024-04-13T03:34:05.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.156+0000 7ff0a0600700 1 -- 172.21.15.16:0/2094918677 wait complete. 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.158+0000 7ff0a0600700 1 Processor -- start 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.158+0000 7ff0a0600700 1 -- start start 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.158+0000 7ff0a0600700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff098075510 0x7ff09819f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.158+0000 7ff0a0600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff098076ae0 0x7ff09819f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.158+0000 7ff0a0600700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff09810f0f0 0x7ff0981a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.158+0000 7ff0a0600700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff098114d00 con 0x7ff098075510 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.158+0000 7ff0a0600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff098114b80 con 0x7ff098076ae0 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.158+0000 7ff0a0600700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ff098114e80 con 0x7ff09810f0f0 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09dc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff098075510 0x7ff09819f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09dc00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff098075510 0x7ff09819f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39984/0 (socket says 172.21.15.16:39984) 2024-04-13T03:34:05.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09dc00700 1 -- 172.21.15.16:0/337021545 learned_addr learned my addr 172.21.15.16:0/337021545 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:05.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09d200700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff098076ae0 0x7ff09819f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:05.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09e600700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff09810f0f0 0x7ff0981a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09dc00700 1 -- 172.21.15.16:0/337021545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff09810f0f0 msgr2=0x7ff0981a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09dc00700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff09810f0f0 0x7ff0981a6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09dc00700 1 -- 172.21.15.16:0/337021545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff098076ae0 msgr2=0x7ff09819f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09dc00700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff098076ae0 0x7ff09819f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09dc00700 1 -- 172.21.15.16:0/337021545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0981a6fe0 con 0x7ff098075510 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.159+0000 7ff09d200700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff098076ae0 0x7ff09819f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.160+0000 7ff09dc00700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff098075510 0x7ff09819f380 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0098e0 tx=0x7ff08c00bb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.160+0000 7ff096000700 1 -- 172.21.15.16:0/337021545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c027070 con 0x7ff098075510 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.160+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0981a7270 con 0x7ff098075510 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.160+0000 7ff096000700 1 -- 172.21.15.16:0/337021545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff08c003d70 con 0x7ff098075510 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.160+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0981a7720 con 0x7ff098075510 2024-04-13T03:34:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.160+0000 7ff096000700 1 -- 172.21.15.16:0/337021545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c031060 con 0x7ff098075510 2024-04-13T03:34:05.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.161+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0980697a0 con 0x7ff098075510 2024-04-13T03:34:05.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.164+0000 7ff096000700 1 -- 172.21.15.16:0/337021545 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff08c02a070 con 0x7ff098075510 2024-04-13T03:34:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.165+0000 7ff096000700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff0800a3c40 0x7ff0800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.165+0000 7ff096000700 1 -- 172.21.15.16:0/337021545 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff08c0d1940 con 0x7ff098075510 2024-04-13T03:34:05.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.165+0000 7ff09d200700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff0800a3c40 0x7ff0800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:05.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.166+0000 7ff09d200700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff0800a3c40 0x7ff0800a6100 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff098076810 tx=0x7ff084009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:05.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.168+0000 7ff096000700 1 -- 172.21.15.16:0/337021545 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff08c013070 con 0x7ff098075510 2024-04-13T03:34:05.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.410+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7ff098062ef0 con 0x7ff0800a3c40 2024-04-13T03:34:05.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.431+0000 7ff096000700 1 -- 172.21.15.16:0/337021545 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7ff098062ef0 con 0x7ff0800a3c40 2024-04-13T03:34:05.433 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-13T03:34:05.433 INFO:teuthology.orchestra.run.smithi016.stdout:mds.a smithi016 running (2m) 41s ago 2m 12.4M - 19.0.0-2884-g218a6914 a4e4811815e4 bb75f63a4c63 2024-04-13T03:34:05.433 INFO:teuthology.orchestra.run.smithi016.stdout:mds.b smithi119 running (2m) 85s ago 2m 25.0M - 19.0.0-2884-g218a6914 a4e4811815e4 d80ab5e311a1 2024-04-13T03:34:05.433 INFO:teuthology.orchestra.run.smithi016.stdout:mds.c smithi188 running (2m) 85s ago 2m 17.1M - 19.0.0-2884-g218a6914 a4e4811815e4 ea3859d0b276 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mds.d smithi016 running (3m) 41s ago 3m 15.1M - 19.0.0-2884-g218a6914 a4e4811815e4 050a364f1231 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mds.e smithi119 running (2m) 85s ago 2m 16.9M - 19.0.0-2884-g218a6914 a4e4811815e4 ae69677435c8 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mds.f smithi188 running (2m) 85s ago 2m 18.5M - 19.0.0-2884-g218a6914 a4e4811815e4 2e1bc20a2d70 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mds.g smithi016 running (2m) 41s ago 2m 12.3M - 19.0.0-2884-g218a6914 a4e4811815e4 8fc36c5f69e5 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mds.h smithi119 running (2m) 85s ago 2m 17.6M - 19.0.0-2884-g218a6914 a4e4811815e4 cb36ac31d992 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mds.i smithi188 running (2m) 85s ago 2m 17.5M - 19.0.0-2884-g218a6914 a4e4811815e4 cc5aff15da9c 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mds.j smithi016 running (3m) 41s ago 3m 15.1M - 19.0.0-2884-g218a6914 a4e4811815e4 1495192ece6d 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mds.k smithi119 running (3m) 85s ago 3m 17.5M - 19.0.0-2884-g218a6914 a4e4811815e4 4e18168fa20f 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mds.l smithi188 running (3m) 85s ago 3m 18.4M - 19.0.0-2884-g218a6914 a4e4811815e4 ad4dc24fcb97 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi016 *:9283,8765 running (11m) 41s ago 11m 516M - 19.0.0-2884-g218a6914 a4e4811815e4 d7830156b6df 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi119 *:8443,8765 running (9m) 85s ago 9m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 9069651ddd42 2024-04-13T03:34:05.434 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.z smithi188 *:8443,8765 running (9m) 85s ago 9m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 6df30516e333 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (11m) 41s ago 11m 50.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 c421eb666bd8 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi119 running (9m) 85s ago 9m 41.0M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 a9319192e0cf 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi188 running (9m) 85s ago 9m 41.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 3b30980af288 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (8m) 41s ago 8m 57.5M 4273M 19.0.0-2884-g218a6914 a4e4811815e4 bafebf2370f7 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi119 running (8m) 85s ago 8m 46.5M 4270M 19.0.0-2884-g218a6914 a4e4811815e4 8891e9f2484b 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi188 running (7m) 85s ago 7m 57.3M 4270M 19.0.0-2884-g218a6914 a4e4811815e4 6f6efe624c04 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (7m) 41s ago 7m 45.2M 4273M 19.0.0-2884-g218a6914 a4e4811815e4 80ad07a7b56e 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi119 running (6m) 85s ago 6m 43.1M 4270M 19.0.0-2884-g218a6914 a4e4811815e4 684bddd74810 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi188 running (6m) 85s ago 6m 45.5M 4270M 19.0.0-2884-g218a6914 a4e4811815e4 49b7a8e94483 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi016 running (5m) 41s ago 5m 42.5M 4273M 19.0.0-2884-g218a6914 a4e4811815e4 4fe2a2c64ea6 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi119 running (5m) 85s ago 5m 57.7M 4270M 19.0.0-2884-g218a6914 a4e4811815e4 09d960de05e5 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 smithi188 running (5m) 85s ago 5m 56.2M 4270M 19.0.0-2884-g218a6914 a4e4811815e4 64d3d1ae48ea 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 smithi016 running (4m) 41s ago 4m 52.4M 4273M 19.0.0-2884-g218a6914 a4e4811815e4 572a1f28c164 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 smithi119 running (3m) 85s ago 3m 55.8M 4270M 19.0.0-2884-g218a6914 a4e4811815e4 664bcdb4be1a 2024-04-13T03:34:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 smithi188 running (3m) 85s ago 3m 41.5M 4270M 19.0.0-2884-g218a6914 a4e4811815e4 93f9e9735781 2024-04-13T03:34:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff0800a3c40 msgr2=0x7ff0800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff0800a3c40 0x7ff0800a6100 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff098076810 tx=0x7ff084009260 comp rx=0 tx=0).stop 2024-04-13T03:34:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff098075510 msgr2=0x7ff09819f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff098075510 0x7ff09819f380 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0098e0 tx=0x7ff08c00bb90 comp rx=0 tx=0).stop 2024-04-13T03:34:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 shutdown_connections 2024-04-13T03:34:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7ff0800a3c40 0x7ff0800a6100 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff09810f0f0 0x7ff0981a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff098075510 0x7ff09819f380 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 --2- 172.21.15.16:0/337021545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff098076ae0 0x7ff09819f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.434+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 >> 172.21.15.16:0/337021545 conn(0x7ff0980fe0b0 msgr2=0x7ff098102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.435+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 shutdown_connections 2024-04-13T03:34:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:05.435+0000 7ff0a0600700 1 -- 172.21.15.16:0/337021545 wait complete. 2024-04-13T03:34:05.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:05 smithi119 ceph-mon[35835]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:05.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:05 smithi016 ceph-mon[31772]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:05.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:05 smithi188 ceph-mon[36398]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:06.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-13T03:34:06.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:06 smithi016 ceph-mon[31772]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:06.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:06 smithi188 ceph-mon[36398]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:06.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:06 smithi119 ceph-mon[35835]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:06.834 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:34:07.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:07 smithi016 ceph-mon[31772]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:34:07.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:07 smithi188 ceph-mon[36398]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:34:07.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:07 smithi119 ceph-mon[35835]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:34:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.179+0000 7f590fe00700 1 -- 172.21.15.16:0/3484238150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5908103660 msgr2=0x7f5908105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.179+0000 7f590fe00700 1 --2- 172.21.15.16:0/3484238150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5908103660 0x7f5908105a50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f58fc0098f0 tx=0x7f58fc009c00 comp rx=0 tx=0).stop 2024-04-13T03:34:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.179+0000 7f590fe00700 1 -- 172.21.15.16:0/3484238150 shutdown_connections 2024-04-13T03:34:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.179+0000 7f590fe00700 1 --2- 172.21.15.16:0/3484238150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59081088c0 0x7f590810acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.179+0000 7f590fe00700 1 --2- 172.21.15.16:0/3484238150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5908103660 0x7f5908105a50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.179+0000 7f590fe00700 1 --2- 172.21.15.16:0/3484238150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5908105f90 0x7f5908108380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.179+0000 7f590fe00700 1 -- 172.21.15.16:0/3484238150 >> 172.21.15.16:0/3484238150 conn(0x7f59080fcfb0 msgr2=0x7f59080ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.179+0000 7f590fe00700 1 -- 172.21.15.16:0/3484238150 shutdown_connections 2024-04-13T03:34:09.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.179+0000 7f590fe00700 1 -- 172.21.15.16:0/3484238150 wait complete. 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.181+0000 7f590fe00700 1 Processor -- start 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.181+0000 7f590fe00700 1 -- start start 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.181+0000 7f590fe00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5908103660 0x7f5908198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.181+0000 7f590fe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5908105f90 0x7f5908199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.181+0000 7f590fe00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59081088c0 0x7f59081a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.181+0000 7f590fe00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f590810e700 con 0x7f5908103660 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.181+0000 7f590fe00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f590810e580 con 0x7f5908105f90 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.181+0000 7f590fe00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f590810e880 con 0x7f59081088c0 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5908103660 0x7f5908198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590ca00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5908105f90 0x7f5908199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590d400700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5908103660 0x7f5908198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33328/0 (socket says 172.21.15.16:33328) 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590d400700 1 -- 172.21.15.16:0/603601847 learned_addr learned my addr 172.21.15.16:0/603601847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:09.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590de00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59081088c0 0x7f59081a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590ca00700 1 -- 172.21.15.16:0/603601847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59081088c0 msgr2=0x7f59081a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590ca00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59081088c0 0x7f59081a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590ca00700 1 -- 172.21.15.16:0/603601847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5908103660 msgr2=0x7f5908198dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590ca00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5908103660 0x7f5908198dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590ca00700 1 -- 172.21.15.16:0/603601847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59081a0a30 con 0x7f5908105f90 2024-04-13T03:34:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.182+0000 7f590ca00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5908105f90 0x7f5908199310 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58f400e9f0 tx=0x7f58f400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:09.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.183+0000 7f5906000700 1 -- 172.21.15.16:0/603601847 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f400cb70 con 0x7f5908105f90 2024-04-13T03:34:09.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.183+0000 7f5906000700 1 -- 172.21.15.16:0/603601847 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f400ee90 con 0x7f5908105f90 2024-04-13T03:34:09.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.183+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59081a0d20 con 0x7f5908105f90 2024-04-13T03:34:09.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.183+0000 7f5906000700 1 -- 172.21.15.16:0/603601847 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f4003df0 con 0x7f5908105f90 2024-04-13T03:34:09.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.183+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59081a1250 con 0x7f5908105f90 2024-04-13T03:34:09.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.184+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59080697a0 con 0x7f5908105f90 2024-04-13T03:34:09.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.187+0000 7f5906000700 1 -- 172.21.15.16:0/603601847 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f58f4004000 con 0x7f5908105f90 2024-04-13T03:34:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.188+0000 7f5906000700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f58f00a3cc0 0x7f58f00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.188+0000 7f5906000700 1 -- 172.21.15.16:0/603601847 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f58f40d0810 con 0x7f5908105f90 2024-04-13T03:34:09.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.188+0000 7f590d400700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f58f00a3cc0 0x7f58f00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:09.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.189+0000 7f590d400700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f58f00a3cc0 0x7f58f00a6180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009680 tx=0x7f58fc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:09.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.192+0000 7f5906000700 1 -- 172.21.15.16:0/603601847 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58f409cdc0 con 0x7f5908105f90 2024-04-13T03:34:09.435 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.433+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f5908062ef0 con 0x7f58f00a3cc0 2024-04-13T03:34:09.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.443+0000 7f5906000700 1 -- 172.21.15.16:0/603601847 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f5908062ef0 con 0x7f58f00a3cc0 2024-04-13T03:34:09.444 INFO:teuthology.orchestra.run.smithi016.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-13T03:34:09.444 INFO:teuthology.orchestra.run.smithi016.stdout:mds.all 12/12 89s ago 3m smithi016=a;smithi016=d;smithi016=g;smithi016=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2024-04-13T03:34:09.444 INFO:teuthology.orchestra.run.smithi016.stdout:mgr 3/3 89s ago 9m smithi016=x;smithi119=y;smithi188=z;count:3 2024-04-13T03:34:09.445 INFO:teuthology.orchestra.run.smithi016.stdout:mon 3/3 89s ago 9m smithi016:172.21.15.16=a;smithi119:172.21.15.119=b;smithi188:172.21.15.188=c;count:3 2024-04-13T03:34:09.445 INFO:teuthology.orchestra.run.smithi016.stdout:osd 12 89s ago - 2024-04-13T03:34:09.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f58f00a3cc0 msgr2=0x7f58f00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:09.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f58f00a3cc0 0x7f58f00a6180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f58fc009680 tx=0x7f58fc019040 comp rx=0 tx=0).stop 2024-04-13T03:34:09.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5908105f90 msgr2=0x7f5908199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:09.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5908105f90 0x7f5908199310 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f58f400e9f0 tx=0x7f58f400ed00 comp rx=0 tx=0).stop 2024-04-13T03:34:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 shutdown_connections 2024-04-13T03:34:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f58f00a3cc0 0x7f58f00a6180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f59081088c0 0x7f59081a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5908103660 0x7f5908198dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 --2- 172.21.15.16:0/603601847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5908105f90 0x7f5908199310 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 >> 172.21.15.16:0/603601847 conn(0x7f59080fcfb0 msgr2=0x7f5908104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 shutdown_connections 2024-04-13T03:34:09.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:09.446+0000 7f590fe00700 1 -- 172.21.15.16:0/603601847 wait complete. 2024-04-13T03:34:09.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:09 smithi016 ceph-mon[31772]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:09.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:09 smithi188 ceph-mon[36398]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:09.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:09 smithi119 ceph-mon[35835]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:10.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-13T03:34:10.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:10 smithi016 ceph-mon[31772]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:10.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:10 smithi188 ceph-mon[36398]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:10.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:10 smithi119 ceph-mon[35835]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:10.877 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:34:11.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:11 smithi016 ceph-mon[31772]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:34:11.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:11 smithi188 ceph-mon[36398]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:34:11.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:11 smithi119 ceph-mon[35835]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:34:13.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.357+0000 7fa1a6200700 1 -- 172.21.15.16:0/933204201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1a010a350 msgr2=0x7fa1a010a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:13.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.357+0000 7fa1a6200700 1 --2- 172.21.15.16:0/933204201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1a010a350 0x7fa1a010a7b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0099c0 tx=0x7fa18c009cd0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.357+0000 7fa1a6200700 1 -- 172.21.15.16:0/933204201 shutdown_connections 2024-04-13T03:34:13.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.357+0000 7fa1a6200700 1 --2- 172.21.15.16:0/933204201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1a010acf0 0x7fa1a01115a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.357+0000 7fa1a6200700 1 --2- 172.21.15.16:0/933204201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a0109150 0x7fa1a0109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.357+0000 7fa1a6200700 1 --2- 172.21.15.16:0/933204201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1a010a350 0x7fa1a010a7b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.357+0000 7fa1a6200700 1 -- 172.21.15.16:0/933204201 >> 172.21.15.16:0/933204201 conn(0x7fa1a0078120 msgr2=0x7fa1a007a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:13.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.357+0000 7fa1a6200700 1 -- 172.21.15.16:0/933204201 shutdown_connections 2024-04-13T03:34:13.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.357+0000 7fa1a6200700 1 -- 172.21.15.16:0/933204201 wait complete. 2024-04-13T03:34:13.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.358+0000 7fa1a6200700 1 Processor -- start 2024-04-13T03:34:13.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.358+0000 7fa1a6200700 1 -- start start 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa1a6200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1a0109150 0x7fa1a019f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa1a6200700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a010a350 0x7fa1a019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa1a6200700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1a010acf0 0x7fa1a01a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa1a6200700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a0114d70 con 0x7fa1a010a350 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa1a6200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a0114bf0 con 0x7fa1a0109150 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa1a6200700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a0114ef0 con 0x7fa1a010acf0 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa19ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a010a350 0x7fa1a019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa19ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a010a350 0x7fa1a019f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33338/0 (socket says 172.21.15.16:33338) 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa19ea00700 1 -- 172.21.15.16:0/2512374735 learned_addr learned my addr 172.21.15.16:0/2512374735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa19f400700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1a0109150 0x7fa1a019f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:13.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.359+0000 7fa19fe00700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1a010acf0 0x7fa1a01a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:13.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa19f400700 1 -- 172.21.15.16:0/2512374735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1a010acf0 msgr2=0x7fa1a01a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa19f400700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1a010acf0 0x7fa1a01a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa19f400700 1 -- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a010a350 msgr2=0x7fa1a019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa19f400700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a010a350 0x7fa1a019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa19f400700 1 -- 172.21.15.16:0/2512374735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a01a6f80 con 0x7fa1a0109150 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa19ea00700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a010a350 0x7fa1a019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa19fe00700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1a010acf0 0x7fa1a01a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa19f400700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1a0109150 0x7fa1a019f320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa19000e9c0 tx=0x7fa19000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa197e00700 1 -- 172.21.15.16:0/2512374735 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19000cbd0 con 0x7fa1a0109150 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa197e00700 1 -- 172.21.15.16:0/2512374735 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa190004500 con 0x7fa1a0109150 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa197e00700 1 -- 172.21.15.16:0/2512374735 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa190003df0 con 0x7fa1a0109150 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.360+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a01a7270 con 0x7fa1a0109150 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.361+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1a01a7800 con 0x7fa1a0109150 2024-04-13T03:34:13.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.362+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1a00697a0 con 0x7fa1a0109150 2024-04-13T03:34:13.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.364+0000 7fa197e00700 1 -- 172.21.15.16:0/2512374735 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa190003f90 con 0x7fa1a0109150 2024-04-13T03:34:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.365+0000 7fa197e00700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa1840a3cc0 0x7fa1840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.365+0000 7fa197e00700 1 -- 172.21.15.16:0/2512374735 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa1900d0cc0 con 0x7fa1a0109150 2024-04-13T03:34:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.365+0000 7fa19ea00700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa1840a3cc0 0x7fa1840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:13.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.366+0000 7fa19ea00700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa1840a3cc0 0x7fa1840a6180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa18c000c00 tx=0x7fa18c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:13.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.368+0000 7fa197e00700 1 -- 172.21.15.16:0/2512374735 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa19009d270 con 0x7fa1a0109150 2024-04-13T03:34:13.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:13 smithi016 ceph-mon[31772]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.604+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fa1a0062ef0 con 0x7fa1840a3cc0 2024-04-13T03:34:13.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.606+0000 7fa197e00700 1 -- 172.21.15.16:0/2512374735 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fa1a0062ef0 con 0x7fa1840a3cc0 2024-04-13T03:34:13.608 INFO:teuthology.orchestra.run.smithi016.stdout:HOST ADDR LABELS STATUS 2024-04-13T03:34:13.608 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 172.21.15.16 2024-04-13T03:34:13.608 INFO:teuthology.orchestra.run.smithi016.stdout:smithi119 172.21.15.119 2024-04-13T03:34:13.608 INFO:teuthology.orchestra.run.smithi016.stdout:smithi188 172.21.15.188 2024-04-13T03:34:13.608 INFO:teuthology.orchestra.run.smithi016.stdout:3 hosts in cluster 2024-04-13T03:34:13.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.610+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa1840a3cc0 msgr2=0x7fa1840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:13.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.610+0000 7fa1a6200700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa1840a3cc0 0x7fa1840a6180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa18c000c00 tx=0x7fa18c005fd0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.610+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1a0109150 msgr2=0x7fa1a019f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:13.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.610+0000 7fa1a6200700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1a0109150 0x7fa1a019f320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa19000e9c0 tx=0x7fa19000ecd0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.611+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 shutdown_connections 2024-04-13T03:34:13.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.611+0000 7fa1a6200700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fa1840a3cc0 0x7fa1840a6180 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.611+0000 7fa1a6200700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1a010acf0 0x7fa1a01a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.611+0000 7fa1a6200700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1a0109150 0x7fa1a019f320 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.611+0000 7fa1a6200700 1 --2- 172.21.15.16:0/2512374735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa1a010a350 0x7fa1a019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:13.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.611+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 >> 172.21.15.16:0/2512374735 conn(0x7fa1a0078120 msgr2=0x7fa1a010d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:13.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.611+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 shutdown_connections 2024-04-13T03:34:13.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:13.611+0000 7fa1a6200700 1 -- 172.21.15.16:0/2512374735 wait complete. 2024-04-13T03:34:13.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:13 smithi188 ceph-mon[36398]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:13.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:13 smithi119 ceph-mon[35835]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:14.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-13T03:34:14.962 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:34:15.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:15 smithi016 ceph-mon[31772]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:15.369 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:15 smithi016 ceph-mon[31772]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:15.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:15 smithi188 ceph-mon[36398]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:15.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:15 smithi188 ceph-mon[36398]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:15.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:15 smithi119 ceph-mon[35835]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:15.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:15 smithi119 ceph-mon[35835]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:17.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.378+0000 7f486c800700 1 -- 172.21.15.16:0/185760667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864075490 msgr2=0x7f4864075910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:17.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.378+0000 7f486c800700 1 --2- 172.21.15.16:0/185760667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864075490 0x7f4864075910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f48540098f0 tx=0x7f4854009c00 comp rx=0 tx=0).stop 2024-04-13T03:34:17.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.378+0000 7f486c800700 1 -- 172.21.15.16:0/185760667 shutdown_connections 2024-04-13T03:34:17.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.378+0000 7f486c800700 1 --2- 172.21.15.16:0/185760667 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f486410aa60 0x7f486410ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:17.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.378+0000 7f486c800700 1 --2- 172.21.15.16:0/185760667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4864076a60 0x7f4864074ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:17.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.378+0000 7f486c800700 1 --2- 172.21.15.16:0/185760667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864075490 0x7f4864075910 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:17.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.378+0000 7f486c800700 1 -- 172.21.15.16:0/185760667 >> 172.21.15.16:0/185760667 conn(0x7f48640fe100 msgr2=0x7f4864100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:17.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.379+0000 7f486c800700 1 -- 172.21.15.16:0/185760667 shutdown_connections 2024-04-13T03:34:17.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.379+0000 7f486c800700 1 -- 172.21.15.16:0/185760667 wait complete. 2024-04-13T03:34:17.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.380+0000 7f486c800700 1 Processor -- start 2024-04-13T03:34:17.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.380+0000 7f486c800700 1 -- start start 2024-04-13T03:34:17.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.380+0000 7f486c800700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4864075490 0x7f486419af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:17.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.380+0000 7f486c800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864076a60 0x7f486419b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:17.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.380+0000 7f486c800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f486410aa60 0x7f48641a2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:17.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.380+0000 7f486c800700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4864110960 con 0x7f4864075490 2024-04-13T03:34:17.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.380+0000 7f486c800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f48641107e0 con 0x7f4864076a60 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.380+0000 7f486c800700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4864110ae0 con 0x7f486410aa60 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864076a60 0x7f486419b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4864075490 0x7f486419af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864076a60 0x7f486419b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:53374/0 (socket says 172.21.15.16:53374) 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4864075490 0x7f486419af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33350/0 (socket says 172.21.15.16:33350) 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869400700 1 -- 172.21.15.16:0/4120356665 learned_addr learned my addr 172.21.15.16:0/4120356665 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f486a800700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f486410aa60 0x7f48641a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869400700 1 -- 172.21.15.16:0/4120356665 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f486410aa60 msgr2=0x7f48641a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869400700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f486410aa60 0x7f48641a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869400700 1 -- 172.21.15.16:0/4120356665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4864075490 msgr2=0x7f486419af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869400700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4864075490 0x7f486419af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869400700 1 -- 172.21.15.16:0/4120356665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48641a2bd0 con 0x7f4864076a60 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.381+0000 7f4869400700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864076a60 0x7f486419b4b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4854009810 tx=0x7f4854005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.382+0000 7f485ea00700 1 -- 172.21.15.16:0/4120356665 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4854005440 con 0x7f4864076a60 2024-04-13T03:34:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.382+0000 7f485ea00700 1 -- 172.21.15.16:0/4120356665 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48540055e0 con 0x7f4864076a60 2024-04-13T03:34:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.382+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48641a2e60 con 0x7f4864076a60 2024-04-13T03:34:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.382+0000 7f485ea00700 1 -- 172.21.15.16:0/4120356665 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4854003bc0 con 0x7f4864076a60 2024-04-13T03:34:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.382+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48641a3250 con 0x7f4864076a60 2024-04-13T03:34:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.383+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48640697a0 con 0x7f4864076a60 2024-04-13T03:34:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.386+0000 7f485ea00700 1 -- 172.21.15.16:0/4120356665 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f485400b560 con 0x7f4864076a60 2024-04-13T03:34:17.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.387+0000 7f485ea00700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f484c0a3c40 0x7f484c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:17.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.387+0000 7f485ea00700 1 -- 172.21.15.16:0/4120356665 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f48540d19c0 con 0x7f4864076a60 2024-04-13T03:34:17.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.387+0000 7f4869e00700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f484c0a3c40 0x7f484c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:17.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.388+0000 7f4869e00700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f484c0a3c40 0x7f484c0a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4858009680 tx=0x7f4858009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:17.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.389+0000 7f485ea00700 1 -- 172.21.15.16:0/4120356665 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f485405eff0 con 0x7f4864076a60 2024-04-13T03:34:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:17 smithi016 ceph-mon[31772]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:34:17.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:17 smithi188 ceph-mon[36398]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:34:17.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.636+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 --> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f4864062ef0 con 0x7f484c0a3c40 2024-04-13T03:34:17.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.638+0000 7f485ea00700 1 -- 172.21.15.16:0/4120356665 <== mgr.14152 v2:172.21.15.16:6800/1721562410 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f4864062ef0 con 0x7f484c0a3c40 2024-04-13T03:34:17.640 INFO:teuthology.orchestra.run.smithi016.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-13T03:34:17.640 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002C480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-04-13T03:34:17.641 INFO:teuthology.orchestra.run.smithi016.stdout:smithi119 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400Y1400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-13T03:34:17.641 INFO:teuthology.orchestra.run.smithi016.stdout:smithi188 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500MP400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-13T03:34:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.641+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f484c0a3c40 msgr2=0x7f484c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.641+0000 7f486c800700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f484c0a3c40 0x7f484c0a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4858009680 tx=0x7f4858009260 comp rx=0 tx=0).stop 2024-04-13T03:34:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.641+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864076a60 msgr2=0x7f486419b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.641+0000 7f486c800700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864076a60 0x7f486419b4b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4854009810 tx=0x7f4854005250 comp rx=0 tx=0).stop 2024-04-13T03:34:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.642+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 shutdown_connections 2024-04-13T03:34:17.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.642+0000 7f486c800700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f484c0a3c40 0x7f484c0a6100 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:17.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.642+0000 7f486c800700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f486410aa60 0x7f48641a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:17.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.642+0000 7f486c800700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4864075490 0x7f486419af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:17.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.642+0000 7f486c800700 1 --2- 172.21.15.16:0/4120356665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4864076a60 0x7f486419b4b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:17.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.642+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 >> 172.21.15.16:0/4120356665 conn(0x7f48640fe100 msgr2=0x7f48641069b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:17.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.642+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 shutdown_connections 2024-04-13T03:34:17.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:17.642+0000 7f486c800700 1 -- 172.21.15.16:0/4120356665 wait complete. 2024-04-13T03:34:17.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:17 smithi119 ceph-mon[35835]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:34:18.567 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-13T03:34:18.577 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T03:34:18.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-13T03:34:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:34:19.471 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:19 smithi016 ceph-mon[31772]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:19.471 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:19 smithi016 ceph-mon[31772]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:19 smithi188 ceph-mon[36398]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:19.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:19 smithi188 ceph-mon[36398]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:19.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:19 smithi119 ceph-mon[35835]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:19.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:19 smithi119 ceph-mon[35835]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:21.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.440+0000 7f9306c00700 1 -- 172.21.15.16:0/469008532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9300103c40 msgr2=0x7f93001040a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:21.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.440+0000 7f9306c00700 1 --2- 172.21.15.16:0/469008532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9300103c40 0x7f93001040a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0099c0 tx=0x7f92ec009cd0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.440+0000 7f9306c00700 1 -- 172.21.15.16:0/469008532 shutdown_connections 2024-04-13T03:34:21.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.440+0000 7f9306c00700 1 --2- 172.21.15.16:0/469008532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93001045e0 0x7f930010ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.440+0000 7f9306c00700 1 --2- 172.21.15.16:0/469008532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9300102a40 0x7f9300102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.440+0000 7f9306c00700 1 --2- 172.21.15.16:0/469008532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9300103c40 0x7f93001040a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.440+0000 7f9306c00700 1 -- 172.21.15.16:0/469008532 >> 172.21.15.16:0/469008532 conn(0x7f93000fdfe0 msgr2=0x7f9300100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:21.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.440+0000 7f9306c00700 1 -- 172.21.15.16:0/469008532 shutdown_connections 2024-04-13T03:34:21.442 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:21 smithi016 ceph-mon[31772]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:34:21.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.440+0000 7f9306c00700 1 -- 172.21.15.16:0/469008532 wait complete. 2024-04-13T03:34:21.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.442+0000 7f9306c00700 1 Processor -- start 2024-04-13T03:34:21.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.442+0000 7f9306c00700 1 -- start start 2024-04-13T03:34:21.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.442+0000 7f9306c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300102a40 0x7f9300198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:21.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.442+0000 7f9306c00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9300103c40 0x7f9300199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:21.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.442+0000 7f9306c00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93001045e0 0x7f93001a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:21.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.442+0000 7f9306c00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f930010e620 con 0x7f9300103c40 2024-04-13T03:34:21.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.442+0000 7f9306c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f930010e4a0 con 0x7f9300102a40 2024-04-13T03:34:21.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.442+0000 7f9306c00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f930010e7a0 con 0x7f93001045e0 2024-04-13T03:34:21.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9304e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9300103c40 0x7f9300199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9305800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300102a40 0x7f9300198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9304e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9300103c40 0x7f9300199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41976/0 (socket says 172.21.15.16:41976) 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9305800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300102a40 0x7f9300198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:52338/0 (socket says 172.21.15.16:52338) 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9304e00700 1 -- 172.21.15.16:0/1827170343 learned_addr learned my addr 172.21.15.16:0/1827170343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9306200700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93001045e0 0x7f93001a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9305800700 1 -- 172.21.15.16:0/1827170343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93001045e0 msgr2=0x7f93001a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9305800700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93001045e0 0x7f93001a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9305800700 1 -- 172.21.15.16:0/1827170343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9300103c40 msgr2=0x7f9300199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9305800700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9300103c40 0x7f9300199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9305800700 1 -- 172.21.15.16:0/1827170343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93001a0840 con 0x7f9300102a40 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9306200700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93001045e0 0x7f93001a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:21.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.443+0000 7f9305800700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300102a40 0x7f9300198be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f92f400ef90 tx=0x7f92f400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:21.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.444+0000 7f92fe000700 1 -- 172.21.15.16:0/1827170343 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f400ca70 con 0x7f9300102a40 2024-04-13T03:34:21.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.444+0000 7f92fe000700 1 -- 172.21.15.16:0/1827170343 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92f400cc10 con 0x7f9300102a40 2024-04-13T03:34:21.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.444+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93001a0b30 con 0x7f9300102a40 2024-04-13T03:34:21.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.444+0000 7f92fe000700 1 -- 172.21.15.16:0/1827170343 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4007500 con 0x7f9300102a40 2024-04-13T03:34:21.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.444+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93001a1020 con 0x7f9300102a40 2024-04-13T03:34:21.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.445+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93000697a0 con 0x7f9300102a40 2024-04-13T03:34:21.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.447+0000 7f92fe000700 1 -- 172.21.15.16:0/1827170343 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f92f4012050 con 0x7f9300102a40 2024-04-13T03:34:21.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.449+0000 7f92fe000700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f92e80a3c40 0x7f92e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:21.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.449+0000 7f92fe000700 1 -- 172.21.15.16:0/1827170343 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f92f40d06b0 con 0x7f9300102a40 2024-04-13T03:34:21.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.449+0000 7f9304e00700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f92e80a3c40 0x7f92e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:21.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.450+0000 7f9304e00700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f92e80a3c40 0x7f92e80a6100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f92ec003790 tx=0x7f92ec003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:21.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.451+0000 7f92fe000700 1 -- 172.21.15.16:0/1827170343 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f92f4010040 con 0x7f9300102a40 2024-04-13T03:34:21.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.735+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f930002dfa0 con 0x7f9300102a40 2024-04-13T03:34:21.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.736+0000 7f92fe000700 1 -- 172.21.15.16:0/1827170343 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3092 (secure 0 0 0) 0x7f92f409cc60 con 0x7f9300102a40 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:e26 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:legacy client fscid: 1 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:Filesystem 'cephfs' (1) 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:fs_name cephfs 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 26 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-13T03:31:37.109492+0000 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-13T03:32:07.796132+0000 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:tableserver 0 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:root 0 2024-04-13T03:34:21.738 INFO:teuthology.orchestra.run.smithi016.stdout:session_timeout 300 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:session_autoclose 300 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:max_file_size 1099511627776 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:max_xattr_size 65536 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:required_client_features {} 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure 0 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:last_failure_osd_epoch 0 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:max_mds 5 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:in 0,1,2,3,4 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:up {0=24587,1=24530,2=24523,3=24505,4=24500} 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:failed 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:damaged 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:stopped 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:data_pools [3,4] 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:metadata_pool 2 2024-04-13T03:34:21.739 INFO:teuthology.orchestra.run.smithi016.stdout:inline_data disabled 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:balancer 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:bal_rank_mask -1 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:standby_count_wanted 0 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:qdb_cluster leader: 24587 members: 24505,24500,24587,24523,24530 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b{0:24587} state up:active seq 3 addr [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.f{0:24493} state up:standby-replay seq 1 addr [v2:172.21.15.188:6834/566226134,v1:172.21.15.188:6835/566226134] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.h{1:24530} state up:active seq 13 addr [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d{1:24472} state up:standby-replay seq 1 addr [v2:172.21.15.16:6836/2782624805,v1:172.21.15.16:6837/2782624805] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.i{2:24523} state up:active seq 13 addr [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.j{2:24470} state up:standby-replay seq 1 addr [v2:172.21.15.16:6834/1177424525,v1:172.21.15.16:6835/1177424525] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c{3:24505} state up:active seq 15 addr [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.k{3:24452} state up:standby-replay seq 1 addr [v2:172.21.15.119:6832/3221738952,v1:172.21.15.119:6833/3221738952] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.e{4:24500} state up:active seq 16 addr [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.740 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.l{4:24433} state up:standby-replay seq 1 addr [v2:172.21.15.188:6832/3805092939,v1:172.21.15.188:6833/3805092939] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:34:21.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:34:21.741 INFO:teuthology.orchestra.run.smithi016.stdout:Standby daemons: 2024-04-13T03:34:21.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:34:21.741 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.g{-1:14457} state up:standby seq 1 addr [v2:172.21.15.16:6838/3959195994,v1:172.21.15.16:6839/3959195994] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.741 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a{-1:14475} state up:standby seq 1 addr [v2:172.21.15.16:6840/701411811,v1:172.21.15.16:6841/701411811] compat {c=[1],r=[1],i=[1fff]}] 2024-04-13T03:34:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.740+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f92e80a3c40 msgr2=0x7f92e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.740+0000 7f9306c00700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f92e80a3c40 0x7f92e80a6100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f92ec003790 tx=0x7f92ec003860 comp rx=0 tx=0).stop 2024-04-13T03:34:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300102a40 msgr2=0x7f9300198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300102a40 0x7f9300198be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f92f400ef90 tx=0x7f92f400c3a0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 shutdown_connections 2024-04-13T03:34:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f92e80a3c40 0x7f92e80a6100 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93001045e0 0x7f93001a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9300102a40 0x7f9300198be0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 --2- 172.21.15.16:0/1827170343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9300103c40 0x7f9300199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 >> 172.21.15.16:0/1827170343 conn(0x7f93000fdfe0 msgr2=0x7f9300106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 shutdown_connections 2024-04-13T03:34:21.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:21.741+0000 7f9306c00700 1 -- 172.21.15.16:0/1827170343 wait complete. 2024-04-13T03:34:21.744 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 26 2024-04-13T03:34:21.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:21 smithi119 ceph-mon[35835]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:34:21.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:21 smithi188 ceph-mon[36398]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:34:22.540 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:22 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1827170343' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-13T03:34:22.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-13T03:34:22.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:22 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1827170343' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-13T03:34:22.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:22 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1827170343' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-13T03:34:23.100 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:34:23.500 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:23 smithi016 ceph-mon[31772]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:23.774 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:23 smithi188 ceph-mon[36398]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:23.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:23 smithi119 ceph-mon[35835]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:25.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.505+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1311484947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8810a350 msgr2=0x7f4d8810a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:25.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.505+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1311484947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8810a350 0x7f4d8810a7b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4d740099c0 tx=0x7f4d74009cd0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.505+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1311484947 shutdown_connections 2024-04-13T03:34:25.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.505+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1311484947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d8810acf0 0x7f4d881115a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.505+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1311484947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d88109150 0x7f4d88109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.505+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1311484947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d8810a350 0x7f4d8810a7b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.505+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1311484947 >> 172.21.15.16:0/1311484947 conn(0x7f4d88078120 msgr2=0x7f4d8807a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:25.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.505+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1311484947 shutdown_connections 2024-04-13T03:34:25.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.505+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1311484947 wait complete. 2024-04-13T03:34:25.507 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:25 smithi016 ceph-mon[31772]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:25.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.506+0000 7f4d8ea00700 1 Processor -- start 2024-04-13T03:34:25.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d8ea00700 1 -- start start 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d8ea00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d88109150 0x7f4d8819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d8ea00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d8810a350 0x7f4d8819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d8ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8810acf0 0x7f4d881a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d8ea00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88114d70 con 0x7f4d88109150 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d8ea00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88114bf0 con 0x7f4d8810acf0 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d8ea00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88114ef0 con 0x7f4d8810a350 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d87e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d88109150 0x7f4d8819f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d87e00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d88109150 0x7f4d8819f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41990/0 (socket says 172.21.15.16:41990) 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.507+0000 7f4d87e00700 1 -- 172.21.15.16:0/1421835715 learned_addr learned my addr 172.21.15.16:0/1421835715 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.508+0000 7f4d8ca00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8810acf0 0x7f4d881a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.508+0000 7f4d87400700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d8810a350 0x7f4d8819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.508+0000 7f4d87e00700 1 -- 172.21.15.16:0/1421835715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d8810a350 msgr2=0x7f4d8819f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.508+0000 7f4d87e00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d8810a350 0x7f4d8819f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.508+0000 7f4d87e00700 1 -- 172.21.15.16:0/1421835715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8810acf0 msgr2=0x7f4d881a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:25.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.508+0000 7f4d87e00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8810acf0 0x7f4d881a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.508+0000 7f4d87e00700 1 -- 172.21.15.16:0/1421835715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d881a6f80 con 0x7f4d88109150 2024-04-13T03:34:25.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.508+0000 7f4d87e00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d88109150 0x7f4d8819f320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800e9a0 tx=0x7f4d7800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:25.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.509+0000 7f4d84c00700 1 -- 172.21.15.16:0/1421835715 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78004220 con 0x7f4d88109150 2024-04-13T03:34:25.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.509+0000 7f4d84c00700 1 -- 172.21.15.16:0/1421835715 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d780043c0 con 0x7f4d88109150 2024-04-13T03:34:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.509+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d881a7270 con 0x7f4d88109150 2024-04-13T03:34:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.509+0000 7f4d84c00700 1 -- 172.21.15.16:0/1421835715 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78003df0 con 0x7f4d88109150 2024-04-13T03:34:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.509+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d881a7800 con 0x7f4d88109150 2024-04-13T03:34:25.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.510+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d880697a0 con 0x7f4d88109150 2024-04-13T03:34:25.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.512+0000 7f4d84c00700 1 -- 172.21.15.16:0/1421835715 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4d7800c810 con 0x7f4d88109150 2024-04-13T03:34:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.514+0000 7f4d84c00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4d6c0a3cc0 0x7f4d6c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.514+0000 7f4d84c00700 1 -- 172.21.15.16:0/1421835715 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4d780d0950 con 0x7f4d88109150 2024-04-13T03:34:25.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.514+0000 7f4d87400700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4d6c0a3cc0 0x7f4d6c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:25.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.515+0000 7f4d87400700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4d6c0a3cc0 0x7f4d6c0a6180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4d74000c00 tx=0x7f4d74005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:25.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.519+0000 7f4d84c00700 1 -- 172.21.15.16:0/1421835715 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d7809cf00 con 0x7f4d88109150 2024-04-13T03:34:25.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.763+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f4d880636c0 con 0x7f4d88109150 2024-04-13T03:34:25.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.765+0000 7f4d84c00700 1 -- 172.21.15.16:0/1421835715 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5713 (secure 0 0 0) 0x7f4d7809c800 con 0x7f4d88109150 2024-04-13T03:34:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 94 2024-04-13T03:34:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:fsid e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:34:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-13T03:22:22.658381+0000 2024-04-13T03:34:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-04-13T03:33:42.153326+0000 2024-04-13T03:34:25.767 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 27 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client luminous 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release squid 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-13T03:34:25.768 INFO:teuthology.orchestra.run.smithi016.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 12 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6802/4119728361,v1:172.21.15.16:6803/4119728361] [v2:172.21.15.16:6804/4119728361,v1:172.21.15.16:6805/4119728361] exists,up 69f3ffdc-4c06-47c2-92c0-1b7aab496918 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 13 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6800/1083610996,v1:172.21.15.119:6801/1083610996] [v2:172.21.15.119:6802/1083610996,v1:172.21.15.119:6803/1083610996] exists,up f603a5d2-28ba-439b-8855-6ff82db896aa 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 18 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6800/1923423179,v1:172.21.15.188:6801/1923423179] [v2:172.21.15.188:6802/1923423179,v1:172.21.15.188:6803/1923423179] exists,up 3c288439-37d1-43e9-8e52-9a08bdbcfff9 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 25 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6810/3993319760,v1:172.21.15.16:6811/3993319760] [v2:172.21.15.16:6812/3993319760,v1:172.21.15.16:6813/3993319760] exists,up a289feb8-b06b-4faf-b5f1-bf88a3cc8b88 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 33 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6808/1038380555,v1:172.21.15.119:6809/1038380555] [v2:172.21.15.119:6810/1038380555,v1:172.21.15.119:6811/1038380555] exists,up d1925505-90c0-4946-b886-25813ecd4c25 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 38 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6808/2068292885,v1:172.21.15.188:6809/2068292885] [v2:172.21.15.188:6810/2068292885,v1:172.21.15.188:6811/2068292885] exists,up c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 up in weight 1 up_from 46 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6818/1633490210,v1:172.21.15.16:6819/1633490210] [v2:172.21.15.16:6820/1633490210,v1:172.21.15.16:6821/1633490210] exists,up facc0570-ab47-4224-9ac9-290d12b6a5f0 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 up in weight 1 up_from 51 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6816/2208117973,v1:172.21.15.119:6817/2208117973] [v2:172.21.15.119:6818/2208117973,v1:172.21.15.119:6819/2208117973] exists,up 61259d55-32c3-438a-bdac-0daf88896936 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.8 up in weight 1 up_from 59 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6816/3338270245,v1:172.21.15.188:6817/3338270245] [v2:172.21.15.188:6818/3338270245,v1:172.21.15.188:6819/3338270245] exists,up 59686f85-07f5-4b22-a3a6-85a29cacda2d 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.9 up in weight 1 up_from 67 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6826/3476141905,v1:172.21.15.16:6827/3476141905] [v2:172.21.15.16:6828/3476141905,v1:172.21.15.16:6829/3476141905] exists,up 19ad4b60-9404-402d-bd53-bcd5a7887f35 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.10 up in weight 1 up_from 73 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6824/1724485593,v1:172.21.15.119:6825/1724485593] [v2:172.21.15.119:6826/1724485593,v1:172.21.15.119:6827/1724485593] exists,up c665a9ff-a65b-4ee2-936d-4169ea0ac9fa 2024-04-13T03:34:25.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.11 up in weight 1 up_from 81 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6824/2606023375,v1:172.21.15.188:6825/2606023375] [v2:172.21.15.188:6826/2606023375,v1:172.21.15.188:6827/2606023375] exists,up 235b52a6-7965-4637-a723-c5802733e041 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.0 [8,4] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.5 [8,11] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.8 [2,11] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 2.14 [2,4] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.2 [3,9] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.12 [3,9] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.1d [4,8] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.31 [4,6] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.34 [4,9] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 3.39 [5,8] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.f [9,8] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.1c [9,8] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.1d [5,8] 2024-04-13T03:34:25.770 INFO:teuthology.orchestra.run.smithi016.stdout:pg_upmap_items 4.1f [9,10] 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/164620940 expires 2024-04-14T03:23:39.714975+0000 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/83197818 expires 2024-04-14T03:23:04.480630+0000 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/20480070 expires 2024-04-14T03:23:39.714975+0000 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3152772162 expires 2024-04-14T03:23:04.480630+0000 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/2135228404 expires 2024-04-14T03:23:04.480630+0000 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/2468290259 expires 2024-04-14T03:23:39.714975+0000 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6801/164620940 expires 2024-04-14T03:23:39.714975+0000 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/1664031642 expires 2024-04-14T03:23:04.480630+0000 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:0/3312707257 expires 2024-04-14T03:23:39.714975+0000 2024-04-13T03:34:25.771 INFO:teuthology.orchestra.run.smithi016.stdout:blocklist 172.21.15.16:6800/2135228404 expires 2024-04-14T03:23:04.480630+0000 2024-04-13T03:34:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.768+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4d6c0a3cc0 msgr2=0x7f4d6c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.768+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4d6c0a3cc0 0x7f4d6c0a6180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4d74000c00 tx=0x7f4d74005fb0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.768+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d88109150 msgr2=0x7f4d8819f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.768+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d88109150 0x7f4d8819f320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800e9a0 tx=0x7f4d7800ecb0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.769+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 shutdown_connections 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.769+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f4d6c0a3cc0 0x7f4d6c0a6180 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.769+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8810acf0 0x7f4d881a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.769+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d88109150 0x7f4d8819f320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.769+0000 7f4d8ea00700 1 --2- 172.21.15.16:0/1421835715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d8810a350 0x7f4d8819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.769+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 >> 172.21.15.16:0/1421835715 conn(0x7f4d88078120 msgr2=0x7f4d8810d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.769+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 shutdown_connections 2024-04-13T03:34:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:25.769+0000 7f4d8ea00700 1 -- 172.21.15.16:0/1421835715 wait complete. 2024-04-13T03:34:25.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:25 smithi119 ceph-mon[35835]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:25.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:25 smithi188 ceph-mon[36398]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:26.513 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:26 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1421835715' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-13T03:34:26.513 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:26 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:26.563 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-13T03:34:26.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:34:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.785+0000 7f90b1400640 1 -- 172.21.15.16:0/3050435930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105df0 msgr2=0x7f90ac10c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.785+0000 7f90b1400640 1 --2- 172.21.15.16:0/3050435930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105df0 0x7f90ac10c680 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009ca0 tx=0x7f90a402c100 comp rx=0 tx=0).stop 2024-04-13T03:34:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.785+0000 7f90b1400640 1 -- 172.21.15.16:0/3050435930 shutdown_connections 2024-04-13T03:34:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.785+0000 7f90b1400640 1 --2- 172.21.15.16:0/3050435930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105df0 0x7f90ac10c680 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.785+0000 7f90b1400640 1 --2- 172.21.15.16:0/3050435930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ac105430 0x7f90ac1058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.785+0000 7f90b1400640 1 --2- 172.21.15.16:0/3050435930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90ac104240 0x7f90ac104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.785+0000 7f90b1400640 1 -- 172.21.15.16:0/3050435930 >> 172.21.15.16:0/3050435930 conn(0x7f90ac0ff7b0 msgr2=0x7f90ac101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.785+0000 7f90b1400640 1 -- 172.21.15.16:0/3050435930 shutdown_connections 2024-04-13T03:34:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.785+0000 7f90b1400640 1 -- 172.21.15.16:0/3050435930 wait complete. 2024-04-13T03:34:26.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.786+0000 7f90b1400640 1 Processor -- start 2024-04-13T03:34:26.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.786+0000 7f90b1400640 1 -- start start 2024-04-13T03:34:26.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.786+0000 7f90b1400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ac104240 0x7f90ac079d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:26.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.786+0000 7f90b1400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105430 0x7f90ac07a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:26.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.786+0000 7f90b1400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90ac105df0 0x7f90ac0768e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:26.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.786+0000 7f90b1400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f90ac10fed0 con 0x7f90ac104240 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.786+0000 7f90b1400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f90ac10fd50 con 0x7f90ac105df0 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.786+0000 7f90b1400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f90ac110050 con 0x7f90ac105430 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105430 0x7f90ac07a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105430 0x7f90ac07a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54402/0 (socket says 172.21.15.16:54402) 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90a1e00640 1 -- 172.21.15.16:0/931403543 learned_addr learned my addr 172.21.15.16:0/931403543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90ab200640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90ac105df0 0x7f90ac0768e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90aa800640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ac104240 0x7f90ac079d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90a1e00640 1 -- 172.21.15.16:0/931403543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90ac105df0 msgr2=0x7f90ac0768e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90a1e00640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90ac105df0 0x7f90ac0768e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90a1e00640 1 -- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ac104240 msgr2=0x7f90ac079d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90a1e00640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ac104240 0x7f90ac079d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.787+0000 7f90a1e00640 1 -- 172.21.15.16:0/931403543 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90ac0771a0 con 0x7f90ac105430 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.788+0000 7f90a1e00640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105430 0x7f90ac07a2b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f909400ea60 tx=0x7f909400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:26.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.788+0000 7f90a3e00640 1 -- 172.21.15.16:0/931403543 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909400cde0 con 0x7f90ac105430 2024-04-13T03:34:26.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.788+0000 7f90a3e00640 1 -- 172.21.15.16:0/931403543 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90940035f0 con 0x7f90ac105430 2024-04-13T03:34:26.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.788+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90ac1a6920 con 0x7f90ac105430 2024-04-13T03:34:26.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.788+0000 7f90a3e00640 1 -- 172.21.15.16:0/931403543 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9094010670 con 0x7f90ac105430 2024-04-13T03:34:26.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.788+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90ac1a6d30 con 0x7f90ac105430 2024-04-13T03:34:26.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.789+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90ac10b560 con 0x7f90ac105430 2024-04-13T03:34:26.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.791+0000 7f90a3e00640 1 -- 172.21.15.16:0/931403543 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f90940037a0 con 0x7f90ac105430 2024-04-13T03:34:26.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.793+0000 7f90a3e00640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f90900a3bb0 0x7f90900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:26.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.793+0000 7f90a3e00640 1 -- 172.21.15.16:0/931403543 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f90940d0ae0 con 0x7f90ac105430 2024-04-13T03:34:26.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.793+0000 7f90aa800640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f90900a3bb0 0x7f90900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:26.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.794+0000 7f90aa800640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f90900a3bb0 0x7f90900a6070 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f90ac105290 tx=0x7f909c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:26.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:26.796+0000 7f90a3e00640 1 -- 172.21.15.16:0/931403543 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f909409d090 con 0x7f90ac105430 2024-04-13T03:34:26.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:26 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1421835715' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-13T03:34:26.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:26 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:26.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:26 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1421835715' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-13T03:34:26.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:26 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.012+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90ac062c60 con 0x7f90ac105430 2024-04-13T03:34:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.013+0000 7f90a3e00640 1 -- 172.21.15.16:0/931403543 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24874 (secure 0 0 0) 0x7f909409c990 con 0x7f90ac105430 2024-04-13T03:34:27.014 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:34:27.014 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":94,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:33:42.153326+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f90900a3bb0 msgr2=0x7f90900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f90900a3bb0 0x7f90900a6070 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f90ac105290 tx=0x7f909c005f90 comp rx=0 tx=0).stop 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105430 msgr2=0x7f90ac07a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105430 0x7f90ac07a2b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f909400ea60 tx=0x7f909400ef20 comp rx=0 tx=0).stop 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 shutdown_connections 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f90900a3bb0 0x7f90900a6070 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90ac105df0 0x7f90ac0768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90ac105430 0x7f90ac07a2b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 --2- 172.21.15.16:0/931403543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90ac104240 0x7f90ac079d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.016+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 >> 172.21.15.16:0/931403543 conn(0x7f90ac0ff7b0 msgr2=0x7f90ac108730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.017+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 shutdown_connections 2024-04-13T03:34:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.017+0000 7f90b1400640 1 -- 172.21.15.16:0/931403543 wait complete. 2024-04-13T03:34:27.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-13T03:31:34.802688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-13T03:31:36.036786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-13T03:31:39.824413+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-13T03:34:27.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:34:27.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.247+0000 7fb21c200640 1 -- 172.21.15.16:0/513279046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214105db0 msgr2=0x7fb21410c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.247+0000 7fb21c200640 1 --2- 172.21.15.16:0/513279046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214105db0 0x7fb21410c640 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009ca0 tx=0x7fb20c02c100 comp rx=0 tx=0).stop 2024-04-13T03:34:27.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.247+0000 7fb21c200640 1 -- 172.21.15.16:0/513279046 shutdown_connections 2024-04-13T03:34:27.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.247+0000 7fb21c200640 1 --2- 172.21.15.16:0/513279046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214105db0 0x7fb21410c640 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.247+0000 7fb21c200640 1 --2- 172.21.15.16:0/513279046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2141053f0 0x7fb214105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.247+0000 7fb21c200640 1 --2- 172.21.15.16:0/513279046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214104200 0x7fb214104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.247+0000 7fb21c200640 1 -- 172.21.15.16:0/513279046 >> 172.21.15.16:0/513279046 conn(0x7fb2140ff7b0 msgr2=0x7fb214101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.247+0000 7fb21c200640 1 -- 172.21.15.16:0/513279046 shutdown_connections 2024-04-13T03:34:27.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.247+0000 7fb21c200640 1 -- 172.21.15.16:0/513279046 wait complete. 2024-04-13T03:34:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.248+0000 7fb21c200640 1 Processor -- start 2024-04-13T03:34:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.248+0000 7fb21c200640 1 -- start start 2024-04-13T03:34:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.248+0000 7fb21c200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214104200 0x7fb214079d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.248+0000 7fb21c200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2141053f0 0x7fb21407a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.248+0000 7fb21c200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214105db0 0x7fb2140768e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.248+0000 7fb21c200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb21410fe90 con 0x7fb214105db0 2024-04-13T03:34:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.248+0000 7fb21c200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb21410fd10 con 0x7fb214104200 2024-04-13T03:34:27.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.248+0000 7fb21c200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb214110010 con 0x7fb2141053f0 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb218e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2141053f0 0x7fb21407a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb21a200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214105db0 0x7fb2140768e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb219800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214104200 0x7fb214079d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb218e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2141053f0 0x7fb21407a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54420/0 (socket says 172.21.15.16:54420) 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb219800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214104200 0x7fb214079d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:52386/0 (socket says 172.21.15.16:52386) 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb218e00640 1 -- 172.21.15.16:0/3640280157 learned_addr learned my addr 172.21.15.16:0/3640280157 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb21a200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214105db0 0x7fb2140768e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42018/0 (socket says 172.21.15.16:42018) 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb219800640 1 -- 172.21.15.16:0/3640280157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2141053f0 msgr2=0x7fb21407a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb219800640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2141053f0 0x7fb21407a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb219800640 1 -- 172.21.15.16:0/3640280157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214105db0 msgr2=0x7fb2140768e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb219800640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214105db0 0x7fb2140768e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb219800640 1 -- 172.21.15.16:0/3640280157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2140771a0 con 0x7fb214104200 2024-04-13T03:34:27.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.249+0000 7fb218e00640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2141053f0 0x7fb21407a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.250+0000 7fb219800640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214104200 0x7fb214079d70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e9c0 tx=0x7fb20800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.250+0000 7fb212000640 1 -- 172.21.15.16:0/3640280157 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20800cd80 con 0x7fb214104200 2024-04-13T03:34:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.250+0000 7fb212000640 1 -- 172.21.15.16:0/3640280157 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb208010910 con 0x7fb214104200 2024-04-13T03:34:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.250+0000 7fb212000640 1 -- 172.21.15.16:0/3640280157 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208018980 con 0x7fb214104200 2024-04-13T03:34:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.250+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2141a6900 con 0x7fb214104200 2024-04-13T03:34:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.250+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2141a6d10 con 0x7fb214104200 2024-04-13T03:34:27.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.252+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb21410b4c0 con 0x7fb214104200 2024-04-13T03:34:27.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.253+0000 7fb212000640 1 -- 172.21.15.16:0/3640280157 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb20801f020 con 0x7fb214104200 2024-04-13T03:34:27.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.254+0000 7fb212000640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb1fc0a3c30 0x7fb1fc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:27.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.255+0000 7fb212000640 1 -- 172.21.15.16:0/3640280157 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb2080d0740 con 0x7fb214104200 2024-04-13T03:34:27.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.255+0000 7fb218e00640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb1fc0a3c30 0x7fb1fc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:27.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.256+0000 7fb218e00640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb1fc0a3c30 0x7fb1fc0a60f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb214077cf0 tx=0x7fb200005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:27.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.259+0000 7fb212000640 1 -- 172.21.15.16:0/3640280157 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb20809ccf0 con 0x7fb214104200 2024-04-13T03:34:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.468+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb214062c60 con 0x7fb214104200 2024-04-13T03:34:27.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.469+0000 7fb212000640 1 -- 172.21.15.16:0/3640280157 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fb20809c5f0 con 0x7fb214104200 2024-04-13T03:34:27.470 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:34:27.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb1fc0a3c30 msgr2=0x7fb1fc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb1fc0a3c30 0x7fb1fc0a60f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb214077cf0 tx=0x7fb200005f90 comp rx=0 tx=0).stop 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214104200 msgr2=0x7fb214079d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214104200 0x7fb214079d70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb20800e9c0 tx=0x7fb20800ee80 comp rx=0 tx=0).stop 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 shutdown_connections 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb1fc0a3c30 0x7fb1fc0a60f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb214105db0 0x7fb2140768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb2141053f0 0x7fb21407a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 --2- 172.21.15.16:0/3640280157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb214104200 0x7fb214079d70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 >> 172.21.15.16:0/3640280157 conn(0x7fb2140ff7b0 msgr2=0x7fb214108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 shutdown_connections 2024-04-13T03:34:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.472+0000 7fb21c200640 1 -- 172.21.15.16:0/3640280157 wait complete. 2024-04-13T03:34:27.492 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-13T03:34:27.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:27 smithi119 ceph-mon[35835]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:34:27.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:27 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:27 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:27 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:27 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:27 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/931403543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:34:27.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:27 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.679 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:27 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:27 smithi188 ceph-mon[36398]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:34:27.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:27 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:27 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:27 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.688 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:27 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.688 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:27 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/931403543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:34:27.688 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:27 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.688 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:27 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.720+0000 7f6af3200640 1 -- 172.21.15.16:0/1670477945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aec1030d0 msgr2=0x7f6aec103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.720+0000 7f6af3200640 1 --2- 172.21.15.16:0/1670477945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aec1030d0 0x7f6aec103550 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009900 tx=0x7f6adc02bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:27.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.720+0000 7f6af3200640 1 -- 172.21.15.16:0/1670477945 shutdown_connections 2024-04-13T03:34:27.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.720+0000 7f6af3200640 1 --2- 172.21.15.16:0/1670477945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6aec103a90 0x7f6aec10a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.720+0000 7f6af3200640 1 --2- 172.21.15.16:0/1670477945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aec1030d0 0x7f6aec103550 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.720+0000 7f6af3200640 1 --2- 172.21.15.16:0/1670477945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aec101f00 0x7f6aec102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.720+0000 7f6af3200640 1 -- 172.21.15.16:0/1670477945 >> 172.21.15.16:0/1670477945 conn(0x7f6aec0fd4c0 msgr2=0x7f6aec0ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:27.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.720+0000 7f6af3200640 1 -- 172.21.15.16:0/1670477945 shutdown_connections 2024-04-13T03:34:27.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.720+0000 7f6af3200640 1 -- 172.21.15.16:0/1670477945 wait complete. 2024-04-13T03:34:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.721+0000 7f6af3200640 1 Processor -- start 2024-04-13T03:34:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.721+0000 7f6af3200640 1 -- start start 2024-04-13T03:34:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af3200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6aec101f00 0x7f6aec19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af3200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aec1030d0 0x7f6aec19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af3200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aec103a90 0x7f6aec1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af3200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec10db80 con 0x7f6aec103a90 2024-04-13T03:34:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af3200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec10da00 con 0x7f6aec101f00 2024-04-13T03:34:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af3200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec10dd00 con 0x7f6aec1030d0 2024-04-13T03:34:27.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6aec101f00 0x7f6aec19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af1400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aec1030d0 0x7f6aec19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6aec101f00 0x7f6aec19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:52410/0 (socket says 172.21.15.16:52410) 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af1e00640 1 -- 172.21.15.16:0/2829932622 learned_addr learned my addr 172.21.15.16:0/2829932622 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af2800640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aec103a90 0x7f6aec1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af1e00640 1 -- 172.21.15.16:0/2829932622 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aec1030d0 msgr2=0x7f6aec19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af1e00640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aec1030d0 0x7f6aec19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af1e00640 1 -- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aec103a90 msgr2=0x7f6aec1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af1e00640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aec103a90 0x7f6aec1a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.722+0000 7f6af1e00640 1 -- 172.21.15.16:0/2829932622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec1a2020 con 0x7f6aec101f00 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.723+0000 7f6af1e00640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6aec101f00 0x7f6aec19a400 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000e9c0 tx=0x7f6ae000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.723+0000 7f6ae6a00640 1 -- 172.21.15.16:0/2829932622 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae000cd80 con 0x7f6aec101f00 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.723+0000 7f6ae6a00640 1 -- 172.21.15.16:0/2829932622 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae00035f0 con 0x7f6aec101f00 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.723+0000 7f6ae6a00640 1 -- 172.21.15.16:0/2829932622 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0010690 con 0x7f6aec101f00 2024-04-13T03:34:27.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.723+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aec1a2310 con 0x7f6aec101f00 2024-04-13T03:34:27.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.723+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aec1a2830 con 0x7f6aec101f00 2024-04-13T03:34:27.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.724+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aec1a2af0 con 0x7f6aec101f00 2024-04-13T03:34:27.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.728+0000 7f6ae6a00640 1 -- 172.21.15.16:0/2829932622 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6ae00037a0 con 0x7f6aec101f00 2024-04-13T03:34:27.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.729+0000 7f6ae6a00640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6ad40a3bb0 0x7f6ad40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:27.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.730+0000 7f6ae6a00640 1 -- 172.21.15.16:0/2829932622 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6ae00d0ce0 con 0x7f6aec101f00 2024-04-13T03:34:27.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.730+0000 7f6af1400640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6ad40a3bb0 0x7f6ad40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:27.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.731+0000 7f6af1400640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6ad40a3bb0 0x7f6ad40a6070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6adc02c240 tx=0x7f6adc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:27.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.733+0000 7f6ae6a00640 1 -- 172.21.15.16:0/2829932622 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ae009d290 con 0x7f6aec101f00 2024-04-13T03:34:27.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:27 smithi016 ceph-mon[31772]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-13T03:34:27.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:27 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:27 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:27 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:27 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:27 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/931403543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:34:27.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:27 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:27 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.952+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6aec1a2cc0 con 0x7f6aec101f00 2024-04-13T03:34:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.953+0000 7f6ae6a00640 1 -- 172.21.15.16:0/2829932622 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f6ae009cb90 con 0x7f6aec101f00 2024-04-13T03:34:27.953 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:34:27.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6ad40a3bb0 msgr2=0x7f6ad40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6ad40a3bb0 0x7f6ad40a6070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6adc02c240 tx=0x7f6adc02df90 comp rx=0 tx=0).stop 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6aec101f00 msgr2=0x7f6aec19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6aec101f00 0x7f6aec19a400 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000e9c0 tx=0x7f6ae000ee80 comp rx=0 tx=0).stop 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 shutdown_connections 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6ad40a3bb0 0x7f6ad40a6070 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6aec103a90 0x7f6aec1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6aec1030d0 0x7f6aec19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 --2- 172.21.15.16:0/2829932622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6aec101f00 0x7f6aec19a400 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 >> 172.21.15.16:0/2829932622 conn(0x7f6aec0fd4c0 msgr2=0x7f6aec0741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.955+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 shutdown_connections 2024-04-13T03:34:27.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:27.956+0000 7f6af3200640 1 -- 172.21.15.16:0/2829932622 wait complete. 2024-04-13T03:34:27.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-13T03:34:28.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.168+0000 7fe6d6200640 1 -- 172.21.15.16:0/3564064392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d0102000 msgr2=0x7fe6d0102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.168+0000 7fe6d6200640 1 --2- 172.21.15.16:0/3564064392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d0102000 0x7fe6d0102420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009900 tx=0x7fe6c002bc40 comp rx=0 tx=0).stop 2024-04-13T03:34:28.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.168+0000 7fe6d6200640 1 -- 172.21.15.16:0/3564064392 shutdown_connections 2024-04-13T03:34:28.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.168+0000 7fe6d6200640 1 --2- 172.21.15.16:0/3564064392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6d0103bb0 0x7fe6d010a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.168+0000 7fe6d6200640 1 --2- 172.21.15.16:0/3564064392 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d01031f0 0x7fe6d0103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.168+0000 7fe6d6200640 1 --2- 172.21.15.16:0/3564064392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d0102000 0x7fe6d0102420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.168+0000 7fe6d6200640 1 -- 172.21.15.16:0/3564064392 >> 172.21.15.16:0/3564064392 conn(0x7fe6d00fd570 msgr2=0x7fe6d00ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:28.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.168+0000 7fe6d6200640 1 -- 172.21.15.16:0/3564064392 shutdown_connections 2024-04-13T03:34:28.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.168+0000 7fe6d6200640 1 -- 172.21.15.16:0/3564064392 wait complete. 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.169+0000 7fe6d6200640 1 Processor -- start 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.169+0000 7fe6d6200640 1 -- start start 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.169+0000 7fe6d6200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6d0102000 0x7fe6d0198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.169+0000 7fe6d6200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d01031f0 0x7fe6d0198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.169+0000 7fe6d6200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d0103bb0 0x7fe6d019f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.169+0000 7fe6d6200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d010dce0 con 0x7fe6d0103bb0 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.169+0000 7fe6d6200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d010db60 con 0x7fe6d01031f0 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.169+0000 7fe6d6200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d010de60 con 0x7fe6d0102000 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d01031f0 0x7fe6d0198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cf400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6d0102000 0x7fe6d0198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d01031f0 0x7fe6d0198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:52424/0 (socket says 172.21.15.16:52424) 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cea00640 1 -- 172.21.15.16:0/421256043 learned_addr learned my addr 172.21.15.16:0/421256043 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cf400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6d0102000 0x7fe6d0198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54464/0 (socket says 172.21.15.16:54464) 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cfe00640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d0103bb0 0x7fe6d019f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cea00640 1 -- 172.21.15.16:0/421256043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6d0102000 msgr2=0x7fe6d0198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cea00640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6d0102000 0x7fe6d0198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cea00640 1 -- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d0103bb0 msgr2=0x7fe6d019f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cea00640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d0103bb0 0x7fe6d019f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cea00640 1 -- 172.21.15.16:0/421256043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d019ff30 con 0x7fe6d01031f0 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cfe00640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d0103bb0 0x7fe6d019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.170+0000 7fe6cea00640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d01031f0 0x7fe6d0198850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00d950 tx=0x7fe6bc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.171+0000 7fe6c7e00640 1 -- 172.21.15.16:0/421256043 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc017070 con 0x7fe6d01031f0 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.171+0000 7fe6c7e00640 1 -- 172.21.15.16:0/421256043 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6bc009c30 con 0x7fe6d01031f0 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.171+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d01a0220 con 0x7fe6d01031f0 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.171+0000 7fe6c7e00640 1 -- 172.21.15.16:0/421256043 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc00f6b0 con 0x7fe6d01031f0 2024-04-13T03:34:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.171+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d01a07c0 con 0x7fe6d01031f0 2024-04-13T03:34:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.172+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d0109320 con 0x7fe6d01031f0 2024-04-13T03:34:28.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.175+0000 7fe6c7e00640 1 -- 172.21.15.16:0/421256043 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe6bc0109d0 con 0x7fe6d01031f0 2024-04-13T03:34:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.176+0000 7fe6c7e00640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6b40a3c30 0x7fe6b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:28.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.176+0000 7fe6c7e00640 1 -- 172.21.15.16:0/421256043 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe6bc0d03d0 con 0x7fe6d01031f0 2024-04-13T03:34:28.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.177+0000 7fe6cf400640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6b40a3c30 0x7fe6b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.178+0000 7fe6cf400640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6b40a3c30 0x7fe6b40a60f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe6c002c180 tx=0x7fe6c002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.181+0000 7fe6c7e00640 1 -- 172.21.15.16:0/421256043 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe6bc009de0 con 0x7fe6d01031f0 2024-04-13T03:34:28.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.402+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe6d0062bc0 con 0x7fe6d01031f0 2024-04-13T03:34:28.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.403+0000 7fe6c7e00640 1 -- 172.21.15.16:0/421256043 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fe6bc09c980 con 0x7fe6d01031f0 2024-04-13T03:34:28.403 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-13T03:34:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6b40a3c30 msgr2=0x7fe6b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6b40a3c30 0x7fe6b40a60f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe6c002c180 tx=0x7fe6c002cf90 comp rx=0 tx=0).stop 2024-04-13T03:34:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d01031f0 msgr2=0x7fe6d0198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d01031f0 0x7fe6d0198850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00d950 tx=0x7fe6bc00de10 comp rx=0 tx=0).stop 2024-04-13T03:34:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 shutdown_connections 2024-04-13T03:34:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fe6b40a3c30 0x7fe6b40a60f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6d0103bb0 0x7fe6d019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d01031f0 0x7fe6d0198850 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 --2- 172.21.15.16:0/421256043 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6d0102000 0x7fe6d0198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 >> 172.21.15.16:0/421256043 conn(0x7fe6d00fd570 msgr2=0x7fe6d01064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 shutdown_connections 2024-04-13T03:34:28.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.406+0000 7fe6d6200640 1 -- 172.21.15.16:0/421256043 wait complete. 2024-04-13T03:34:28.428 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-13T03:34:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.665+0000 7fc40fe00640 1 -- 172.21.15.16:0/705731007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc410101f00 msgr2=0x7fc410102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.665+0000 7fc40fe00640 1 --2- 172.21.15.16:0/705731007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc410101f00 0x7fc410102340 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc400009900 tx=0x7fc40002bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.665+0000 7fc40fe00640 1 -- 172.21.15.16:0/705731007 shutdown_connections 2024-04-13T03:34:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.665+0000 7fc40fe00640 1 --2- 172.21.15.16:0/705731007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc410103a90 0x7fc41010a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.665+0000 7fc40fe00640 1 --2- 172.21.15.16:0/705731007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4101030d0 0x7fc410103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.665+0000 7fc40fe00640 1 --2- 172.21.15.16:0/705731007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc410101f00 0x7fc410102340 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.665+0000 7fc40fe00640 1 -- 172.21.15.16:0/705731007 >> 172.21.15.16:0/705731007 conn(0x7fc4100fd4c0 msgr2=0x7fc4100ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.665+0000 7fc40fe00640 1 -- 172.21.15.16:0/705731007 shutdown_connections 2024-04-13T03:34:28.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.665+0000 7fc40fe00640 1 -- 172.21.15.16:0/705731007 wait complete. 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.666+0000 7fc40fe00640 1 Processor -- start 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.666+0000 7fc40fe00640 1 -- start start 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410101f00 0x7fc41019a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40fe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4101030d0 0x7fc41019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc410103a90 0x7fc4101a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40fe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc41010db80 con 0x7fc410101f00 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc41010da00 con 0x7fc410103a90 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40fe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc41010dd00 con 0x7fc4101030d0 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4101030d0 0x7fc41019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc410103a90 0x7fc4101a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40e000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4101030d0 0x7fc41019a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54486/0 (socket says 172.21.15.16:54486) 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc410103a90 0x7fc4101a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:52432/0 (socket says 172.21.15.16:52432) 2024-04-13T03:34:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40e000640 1 -- 172.21.15.16:0/3517627601 learned_addr learned my addr 172.21.15.16:0/3517627601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.667+0000 7fc40ea00640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410101f00 0x7fc41019a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc40f400640 1 -- 172.21.15.16:0/3517627601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4101030d0 msgr2=0x7fc41019a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc40f400640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4101030d0 0x7fc41019a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc40f400640 1 -- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410101f00 msgr2=0x7fc41019a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc40f400640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410101f00 0x7fc41019a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc40f400640 1 -- 172.21.15.16:0/3517627601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4101a2080 con 0x7fc410103a90 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc40e000640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4101030d0 0x7fc41019a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc40f400640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc410103a90 0x7fc4101a1a10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc40800ec50 tx=0x7fc40800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc407400640 1 -- 172.21.15.16:0/3517627601 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408003b70 con 0x7fc410103a90 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc407400640 1 -- 172.21.15.16:0/3517627601 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc408003d10 con 0x7fc410103a90 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4101a2370 con 0x7fc410103a90 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc407400640 1 -- 172.21.15.16:0/3517627601 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408010970 con 0x7fc410103a90 2024-04-13T03:34:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.668+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4101a2870 con 0x7fc410103a90 2024-04-13T03:34:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.670+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4101a2b30 con 0x7fc410103a90 2024-04-13T03:34:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.671+0000 7fc407400640 1 -- 172.21.15.16:0/3517627601 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc408003ec0 con 0x7fc410103a90 2024-04-13T03:34:28.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.673+0000 7fc407400640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc3f40a3bb0 0x7fc3f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:28.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.673+0000 7fc407400640 1 -- 172.21.15.16:0/3517627601 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc4080d0cf0 con 0x7fc410103a90 2024-04-13T03:34:28.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.673+0000 7fc40ea00640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc3f40a3bb0 0x7fc3f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:28.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.674+0000 7fc40ea00640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc3f40a3bb0 0x7fc3f40a6070 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc40002c170 tx=0x7fc40002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:28.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.677+0000 7fc407400640 1 -- 172.21.15.16:0/3517627601 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc40809d300 con 0x7fc410103a90 2024-04-13T03:34:28.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:28 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3640280157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:34:28.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:28 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2829932622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:34:28.679 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:28 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/421256043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:34:28.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:28 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3640280157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:34:28.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:28 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2829932622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:34:28.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:28 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/421256043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:34:28.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:28 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3640280157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:34:28.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:28 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2829932622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:34:28.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:28 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/421256043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:34:28.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.896+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc4101a2d00 con 0x7fc410103a90 2024-04-13T03:34:28.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.897+0000 7fc407400640 1 -- 172.21.15.16:0/3517627601 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fc40809cc00 con 0x7fc410103a90 2024-04-13T03:34:28.898 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:34:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc3f40a3bb0 msgr2=0x7fc3f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc3f40a3bb0 0x7fc3f40a6070 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc40002c170 tx=0x7fc40002df90 comp rx=0 tx=0).stop 2024-04-13T03:34:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc410103a90 msgr2=0x7fc4101a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc410103a90 0x7fc4101a1a10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc40800ec50 tx=0x7fc40800c6a0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 shutdown_connections 2024-04-13T03:34:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fc3f40a3bb0 0x7fc3f40a6070 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc410103a90 0x7fc4101a1a10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4101030d0 0x7fc41019a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 --2- 172.21.15.16:0/3517627601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc410101f00 0x7fc41019a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 >> 172.21.15.16:0/3517627601 conn(0x7fc4100fd4c0 msgr2=0x7fc4100741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 shutdown_connections 2024-04-13T03:34:28.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:28.900+0000 7fc40fe00640 1 -- 172.21.15.16:0/3517627601 wait complete. 2024-04-13T03:34:28.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:34:29.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.151+0000 7f5f08800640 1 -- 172.21.15.16:0/3550902261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f00105b90 msgr2=0x7f5f00106010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.151+0000 7f5f08800640 1 --2- 172.21.15.16:0/3550902261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f00105b90 0x7f5f00106010 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009ca0 tx=0x7f5efc02c100 comp rx=0 tx=0).stop 2024-04-13T03:34:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.151+0000 7f5f08800640 1 -- 172.21.15.16:0/3550902261 shutdown_connections 2024-04-13T03:34:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.151+0000 7f5f08800640 1 --2- 172.21.15.16:0/3550902261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f00105b90 0x7f5f00106010 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.151+0000 7f5f08800640 1 --2- 172.21.15.16:0/3550902261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f001031e0 0x7f5f00105650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.151+0000 7f5f08800640 1 --2- 172.21.15.16:0/3550902261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f00100870 0x7f5f00102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.151+0000 7f5f08800640 1 -- 172.21.15.16:0/3550902261 >> 172.21.15.16:0/3550902261 conn(0x7f5f000fc470 msgr2=0x7f5f000fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.151+0000 7f5f08800640 1 -- 172.21.15.16:0/3550902261 shutdown_connections 2024-04-13T03:34:29.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.151+0000 7f5f08800640 1 -- 172.21.15.16:0/3550902261 wait complete. 2024-04-13T03:34:29.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.152+0000 7f5f08800640 1 Processor -- start 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.152+0000 7f5f08800640 1 -- start start 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f08800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100870 0x7f5f0019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f08800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f001031e0 0x7f5f0019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f08800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f00105b90 0x7f5f001a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f08800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5f00114260 con 0x7f5f00100870 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f08800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5f001140e0 con 0x7f5f00105b90 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f08800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f5f001143e0 con 0x7f5f001031e0 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f05e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100870 0x7f5f0019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f05e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100870 0x7f5f0019e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38782/0 (socket says 172.21.15.16:38782) 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f05400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f001031e0 0x7f5f0019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f05e00640 1 -- 172.21.15.16:0/1442888116 learned_addr learned my addr 172.21.15.16:0/1442888116 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:29.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f06800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f00105b90 0x7f5f001a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f06800640 1 -- 172.21.15.16:0/1442888116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f001031e0 msgr2=0x7f5f0019ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f06800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f001031e0 0x7f5f0019ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.153+0000 7f5f06800640 1 -- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100870 msgr2=0x7f5f0019e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.154+0000 7f5f06800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100870 0x7f5f0019e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.154+0000 7f5f06800640 1 -- 172.21.15.16:0/1442888116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f001a64c0 con 0x7f5f00105b90 2024-04-13T03:34:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.154+0000 7f5f05e00640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100870 0x7f5f0019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:29.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.154+0000 7f5f06800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f00105b90 0x7f5f001a5e50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5efc037040 tx=0x7f5efc035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.154+0000 7f5efaa00640 1 -- 172.21.15.16:0/1442888116 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc039040 con 0x7f5f00105b90 2024-04-13T03:34:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.154+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f001a6750 con 0x7f5f00105b90 2024-04-13T03:34:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.154+0000 7f5efaa00640 1 -- 172.21.15.16:0/1442888116 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5efc044070 con 0x7f5f00105b90 2024-04-13T03:34:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.154+0000 7f5efaa00640 1 -- 172.21.15.16:0/1442888116 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc004350 con 0x7f5f00105b90 2024-04-13T03:34:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.154+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f001a6c60 con 0x7f5f00105b90 2024-04-13T03:34:29.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.156+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f000fdf60 con 0x7f5f00105b90 2024-04-13T03:34:29.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.157+0000 7f5efaa00640 1 -- 172.21.15.16:0/1442888116 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5efc0044f0 con 0x7f5f00105b90 2024-04-13T03:34:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.159+0000 7f5efaa00640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5ee80a3bb0 0x7f5ee80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.159+0000 7f5efaa00640 1 -- 172.21.15.16:0/1442888116 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5efc0f2d80 con 0x7f5f00105b90 2024-04-13T03:34:29.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.159+0000 7f5f05e00640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5ee80a3bb0 0x7f5ee80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:29.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.160+0000 7f5f05e00640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5ee80a3bb0 0x7f5ee80a6070 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40095f0 tx=0x7f5ef4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:29.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.163+0000 7f5efaa00640 1 -- 172.21.15.16:0/1442888116 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5efc0bf330 con 0x7f5f00105b90 2024-04-13T03:34:29.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.438+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f00062bc0 con 0x7f5f00105b90 2024-04-13T03:34:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.439+0000 7f5efaa00640 1 -- 172.21.15.16:0/1442888116 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11934 (secure 0 0 0) 0x7f5efc0098b0 con 0x7f5f00105b90 2024-04-13T03:34:29.439 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:34:29.439 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:32:07.796132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24530,"mds_2":24523,"mds_3":24505,"mds_4":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24505,24500,24587,24523,24530]},"id":1}]} 2024-04-13T03:34:29.439 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 26 2024-04-13T03:34:29.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5ee80a3bb0 msgr2=0x7f5ee80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5ee80a3bb0 0x7f5ee80a6070 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40095f0 tx=0x7f5ef4005f90 comp rx=0 tx=0).stop 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f00105b90 msgr2=0x7f5f001a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f00105b90 0x7f5f001a5e50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5efc037040 tx=0x7f5efc035d10 comp rx=0 tx=0).stop 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 shutdown_connections 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f5ee80a3bb0 0x7f5ee80a6070 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f00105b90 0x7f5f001a5e50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f001031e0 0x7f5f0019ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 --2- 172.21.15.16:0/1442888116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f00100870 0x7f5f0019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 >> 172.21.15.16:0/1442888116 conn(0x7f5f000fc470 msgr2=0x7f5f001011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 shutdown_connections 2024-04-13T03:34:29.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.442+0000 7f5f08800640 1 -- 172.21.15.16:0/1442888116 wait complete. 2024-04-13T03:34:29.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:34:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.665+0000 7f0df7e00640 1 -- 172.21.15.16:0/3616422729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0df8101a30 msgr2=0x7f0df8101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.665+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3616422729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0df8101a30 0x7f0df8101eb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009be0 tx=0x7f0df002bf80 comp rx=0 tx=0).stop 2024-04-13T03:34:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.665+0000 7f0df7e00640 1 -- 172.21.15.16:0/3616422729 shutdown_connections 2024-04-13T03:34:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.665+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3616422729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0df8101a30 0x7f0df8101eb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.665+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3616422729 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0df8101090 0x7f0df81014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.665+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3616422729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8100730 0x7f0df8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.665+0000 7f0df7e00640 1 -- 172.21.15.16:0/3616422729 >> 172.21.15.16:0/3616422729 conn(0x7f0df80fc290 msgr2=0x7f0df80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.665+0000 7f0df7e00640 1 -- 172.21.15.16:0/3616422729 shutdown_connections 2024-04-13T03:34:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.665+0000 7f0df7e00640 1 -- 172.21.15.16:0/3616422729 wait complete. 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.666+0000 7f0df7e00640 1 Processor -- start 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df7e00640 1 -- start start 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0df8100730 0x7f0df819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df7e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8101090 0x7f0df819a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0df8101a30 0x7f0df81a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df7e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0df810fcd0 con 0x7f0df8101090 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df7e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0df810fb50 con 0x7f0df8100730 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df7e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0df810fe50 con 0x7f0df8101a30 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8101090 0x7f0df819a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0df8101a30 0x7f0df81a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df6000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8101090 0x7f0df819a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38814/0 (socket says 172.21.15.16:38814) 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df7400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0df8101a30 0x7f0df81a18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53836/0 (socket says 172.21.15.16:53836) 2024-04-13T03:34:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df6000640 1 -- 172.21.15.16:0/3878042797 learned_addr learned my addr 172.21.15.16:0/3878042797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.667+0000 7f0df6a00640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0df8100730 0x7f0df819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0df7400640 1 -- 172.21.15.16:0/3878042797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0df8100730 msgr2=0x7f0df819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0df7400640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0df8100730 0x7f0df819a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0df7400640 1 -- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8101090 msgr2=0x7f0df819a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0df7400640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8101090 0x7f0df819a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0df7400640 1 -- 172.21.15.16:0/3878042797 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df81a1f20 con 0x7f0df8101a30 2024-04-13T03:34:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0df6000640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8101090 0x7f0df819a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0df7400640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0df8101a30 0x7f0df81a18b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0df0036040 tx=0x7f0df0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0def400640 1 -- 172.21.15.16:0/3878042797 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0043020 con 0x7f0df8101a30 2024-04-13T03:34:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0def400640 1 -- 172.21.15.16:0/3878042797 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0df0004670 con 0x7f0df8101a30 2024-04-13T03:34:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df81a21b0 con 0x7f0df8101a30 2024-04-13T03:34:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.668+0000 7f0def400640 1 -- 172.21.15.16:0/3878042797 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0034690 con 0x7f0df8101a30 2024-04-13T03:34:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.669+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df81a2750 con 0x7f0df8101a30 2024-04-13T03:34:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.670+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0df8068d00 con 0x7f0df8101a30 2024-04-13T03:34:29.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.672+0000 7f0def400640 1 -- 172.21.15.16:0/3878042797 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0df0046020 con 0x7f0df8101a30 2024-04-13T03:34:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.674+0000 7f0def400640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ddc0a3c30 0x7f0ddc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.674+0000 7f0def400640 1 -- 172.21.15.16:0/3878042797 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0df00f2ef0 con 0x7f0df8101a30 2024-04-13T03:34:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.674+0000 7f0df6a00640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ddc0a3c30 0x7f0ddc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.675+0000 7f0df6a00640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ddc0a3c30 0x7f0ddc0a60f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0de8005fd0 tx=0x7f0de8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.678+0000 7f0def400640 1 -- 172.21.15.16:0/3878042797 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0df004d020 con 0x7f0df8101a30 2024-04-13T03:34:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:29 smithi016 ceph-mon[31772]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:29 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:29 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3517627601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:34:29.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:29 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1442888116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:34:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:29 smithi188 ceph-mon[36398]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:29 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:29 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3517627601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:34:29.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:29 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1442888116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:34:29.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.919+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0df8062d80 con 0x7f0df8101a30 2024-04-13T03:34:29.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.921+0000 7f0def400640 1 -- 172.21.15.16:0/3878042797 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24874 (secure 0 0 0) 0x7f0df00bf4a0 con 0x7f0df8101a30 2024-04-13T03:34:29.922 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:34:29.922 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":94,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:33:42.153326+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:34:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ddc0a3c30 msgr2=0x7f0ddc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ddc0a3c30 0x7f0ddc0a60f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0de8005fd0 tx=0x7f0de8009260 comp rx=0 tx=0).stop 2024-04-13T03:34:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0df8101a30 msgr2=0x7f0df81a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0df8101a30 0x7f0df81a18b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0df0036040 tx=0x7f0df0004270 comp rx=0 tx=0).stop 2024-04-13T03:34:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 shutdown_connections 2024-04-13T03:34:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0ddc0a3c30 0x7f0ddc0a60f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0df8101a30 0x7f0df81a18b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0df8101090 0x7f0df819a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 --2- 172.21.15.16:0/3878042797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0df8100730 0x7f0df819a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:29.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 >> 172.21.15.16:0/3878042797 conn(0x7f0df80fc290 msgr2=0x7f0df80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:29.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.925+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 shutdown_connections 2024-04-13T03:34:29.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:29.926+0000 7f0df7e00640 1 -- 172.21.15.16:0/3878042797 wait complete. 2024-04-13T03:34:29.945 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-13T03:31:34.802688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-13T03:31:36.036786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-13T03:31:39.824413+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-13T03:34:29.946 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:34:30.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:29 smithi119 ceph-mon[35835]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:30.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:30.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:30.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:30.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:29 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:30.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:29 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3517627601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:34:30.074 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:29 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1442888116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:34:30.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.165+0000 7fdef7000640 1 -- 172.21.15.16:0/3697173010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdef0102030 msgr2=0x7fdef0102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.165+0000 7fdef7000640 1 --2- 172.21.15.16:0/3697173010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdef0102030 0x7fdef0102450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009900 tx=0x7fdee002bc40 comp rx=0 tx=0).stop 2024-04-13T03:34:30.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.166+0000 7fdef7000640 1 -- 172.21.15.16:0/3697173010 shutdown_connections 2024-04-13T03:34:30.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.166+0000 7fdef7000640 1 --2- 172.21.15.16:0/3697173010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0103be0 0x7fdef010a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.166+0000 7fdef7000640 1 --2- 172.21.15.16:0/3697173010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdef0103220 0x7fdef01036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.166+0000 7fdef7000640 1 --2- 172.21.15.16:0/3697173010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdef0102030 0x7fdef0102450 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.166+0000 7fdef7000640 1 -- 172.21.15.16:0/3697173010 >> 172.21.15.16:0/3697173010 conn(0x7fdef00fd580 msgr2=0x7fdef00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.166+0000 7fdef7000640 1 -- 172.21.15.16:0/3697173010 shutdown_connections 2024-04-13T03:34:30.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.166+0000 7fdef7000640 1 -- 172.21.15.16:0/3697173010 wait complete. 2024-04-13T03:34:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.167+0000 7fdef7000640 1 Processor -- start 2024-04-13T03:34:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.167+0000 7fdef7000640 1 -- start start 2024-04-13T03:34:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.167+0000 7fdef7000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdef0102030 0x7fdef0198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:30.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.167+0000 7fdef7000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0103220 0x7fdef0198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.167+0000 7fdef7000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdef0103be0 0x7fdef019f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.167+0000 7fdef7000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdef010dd10 con 0x7fdef0103220 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.167+0000 7fdef7000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdef010db90 con 0x7fdef0102030 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.167+0000 7fdef7000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fdef010de90 con 0x7fdef0103be0 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeef400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0103220 0x7fdef0198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeefe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdef0102030 0x7fdef0198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeef400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0103220 0x7fdef0198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38840/0 (socket says 172.21.15.16:38840) 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeef400640 1 -- 172.21.15.16:0/2357402261 learned_addr learned my addr 172.21.15.16:0/2357402261 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdef5000640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdef0103be0 0x7fdef019f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeef400640 1 -- 172.21.15.16:0/2357402261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdef0103be0 msgr2=0x7fdef019f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeef400640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdef0103be0 0x7fdef019f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeef400640 1 -- 172.21.15.16:0/2357402261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdef0102030 msgr2=0x7fdef0198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeef400640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdef0102030 0x7fdef0198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeef400640 1 -- 172.21.15.16:0/2357402261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef019ff50 con 0x7fdef0103220 2024-04-13T03:34:30.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.168+0000 7fdeefe00640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdef0102030 0x7fdef0198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.169+0000 7fdeef400640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0103220 0x7fdef0198870 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00e9f0 tx=0x7fdedc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.169+0000 7fdeecc00640 1 -- 172.21.15.16:0/2357402261 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc00cd80 con 0x7fdef0103220 2024-04-13T03:34:30.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.169+0000 7fdeecc00640 1 -- 172.21.15.16:0/2357402261 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdedc005500 con 0x7fdef0103220 2024-04-13T03:34:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.169+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef01a0240 con 0x7fdef0103220 2024-04-13T03:34:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.169+0000 7fdeecc00640 1 -- 172.21.15.16:0/2357402261 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc010690 con 0x7fdef0103220 2024-04-13T03:34:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.169+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef01a0740 con 0x7fdef0103220 2024-04-13T03:34:30.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.170+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef0109350 con 0x7fdef0103220 2024-04-13T03:34:30.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.173+0000 7fdeecc00640 1 -- 172.21.15.16:0/2357402261 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdedc0046c0 con 0x7fdef0103220 2024-04-13T03:34:30.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.174+0000 7fdeecc00640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fded40a3bb0 0x7fded40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.175+0000 7fdeecc00640 1 -- 172.21.15.16:0/2357402261 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fdedc0d0c30 con 0x7fdef0103220 2024-04-13T03:34:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.175+0000 7fdeefe00640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fded40a3bb0 0x7fded40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:30.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.175+0000 7fdeefe00640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fded40a3bb0 0x7fded40a6070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdee002c180 tx=0x7fdee002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:30.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.181+0000 7fdeecc00640 1 -- 172.21.15.16:0/2357402261 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdedc09d1e0 con 0x7fdef0103220 2024-04-13T03:34:30.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.390+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdef0062bc0 con 0x7fdef0103220 2024-04-13T03:34:30.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.392+0000 7fdeecc00640 1 -- 172.21.15.16:0/2357402261 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fdedc09cae0 con 0x7fdef0103220 2024-04-13T03:34:30.392 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:34:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fded40a3bb0 msgr2=0x7fded40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fded40a3bb0 0x7fded40a6070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdee002c180 tx=0x7fdee002cf90 comp rx=0 tx=0).stop 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0103220 msgr2=0x7fdef0198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0103220 0x7fdef0198870 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdedc00e9f0 tx=0x7fdedc00edf0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 shutdown_connections 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fded40a3bb0 0x7fded40a6070 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdef0103be0 0x7fdef019f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdef0103220 0x7fdef0198870 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 --2- 172.21.15.16:0/2357402261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdef0102030 0x7fdef0198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 >> 172.21.15.16:0/2357402261 conn(0x7fdef00fd580 msgr2=0x7fdef0106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 shutdown_connections 2024-04-13T03:34:30.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.395+0000 7fdef7000640 1 -- 172.21.15.16:0/2357402261 wait complete. 2024-04-13T03:34:30.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-13T03:34:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.637+0000 7f8ffe200640 1 -- 172.21.15.16:0/462505152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff8101fc0 msgr2=0x7f8ff8102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.637+0000 7f8ffe200640 1 --2- 172.21.15.16:0/462505152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff8101fc0 0x7f8ff8102440 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009900 tx=0x7f8fe402bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.637+0000 7f8ffe200640 1 -- 172.21.15.16:0/462505152 shutdown_connections 2024-04-13T03:34:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.637+0000 7f8ffe200640 1 --2- 172.21.15.16:0/462505152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ff8102980 0x7f8ff8109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.637+0000 7f8ffe200640 1 --2- 172.21.15.16:0/462505152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff8101fc0 0x7f8ff8102440 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.637+0000 7f8ffe200640 1 --2- 172.21.15.16:0/462505152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8100dd0 0x7f8ff81011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.637+0000 7f8ffe200640 1 -- 172.21.15.16:0/462505152 >> 172.21.15.16:0/462505152 conn(0x7f8ff80fc360 msgr2=0x7f8ff80fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:30.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.638+0000 7f8ffe200640 1 -- 172.21.15.16:0/462505152 shutdown_connections 2024-04-13T03:34:30.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.638+0000 7f8ffe200640 1 -- 172.21.15.16:0/462505152 wait complete. 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffe200640 1 Processor -- start 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffe200640 1 -- start start 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffe200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8100dd0 0x7f8ff810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffe200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ff8101fc0 0x7f8ff810bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffe200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff8102980 0x7f8ff810c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffe200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8069e00 con 0x7f8ff8100dd0 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffe200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8069c80 con 0x7f8ff8102980 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffe200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8069f80 con 0x7f8ff8101fc0 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8100dd0 0x7f8ff810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffce00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8100dd0 0x7f8ff810d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38846/0 (socket says 172.21.15.16:38846) 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffce00640 1 -- 172.21.15.16:0/2264308772 learned_addr learned my addr 172.21.15.16:0/2264308772 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.639+0000 7f8ffd800640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff8102980 0x7f8ff810c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.640+0000 7f8ff7e00640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ff8101fc0 0x7f8ff810bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:30.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.640+0000 7f8ffd800640 1 -- 172.21.15.16:0/2264308772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ff8101fc0 msgr2=0x7f8ff810bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.640+0000 7f8ffd800640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ff8101fc0 0x7f8ff810bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.640+0000 7f8ffd800640 1 -- 172.21.15.16:0/2264308772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8100dd0 msgr2=0x7f8ff810d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.640+0000 7f8ffd800640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8100dd0 0x7f8ff810d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.640+0000 7f8ffd800640 1 -- 172.21.15.16:0/2264308772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff810cd80 con 0x7f8ff8102980 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.640+0000 7f8ffd800640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff8102980 0x7f8ff810c4f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009fd0 tx=0x7f8ff000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.640+0000 7f8ff5600640 1 -- 172.21.15.16:0/2264308772 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff000ce20 con 0x7f8ff8102980 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.641+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81a22b0 con 0x7f8ff8102980 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.641+0000 7f8ff5600640 1 -- 172.21.15.16:0/2264308772 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ff0005020 con 0x7f8ff8102980 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.641+0000 7f8ff5600640 1 -- 172.21.15.16:0/2264308772 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0010720 con 0x7f8ff8102980 2024-04-13T03:34:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.641+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a27d0 con 0x7f8ff8102980 2024-04-13T03:34:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.642+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff8068d00 con 0x7f8ff8102980 2024-04-13T03:34:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.644+0000 7f8ff5600640 1 -- 172.21.15.16:0/2264308772 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8ff0005540 con 0x7f8ff8102980 2024-04-13T03:34:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.645+0000 7f8ff5600640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.645+0000 7f8ff5600640 1 -- 172.21.15.16:0/2264308772 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8ff00d0e80 con 0x7f8ff8102980 2024-04-13T03:34:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.646+0000 7f8ffce00640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.646+0000 7f8ffce00640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0058c0 tx=0x7f8fec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:30.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.649+0000 7f8ff5600640 1 -- 172.21.15.16:0/2264308772 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ff009d430 con 0x7f8ff8102980 2024-04-13T03:34:30.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:30 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3878042797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:34:30.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:30.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:30 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:30.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:30 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2357402261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:34:30.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.869+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8ff8062bc0 con 0x7f8ff8102980 2024-04-13T03:34:30.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.870+0000 7f8ff5600640 1 -- 172.21.15.16:0/2264308772 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f8ff009cd30 con 0x7f8ff8102980 2024-04-13T03:34:30.870 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:34:30.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:30 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3878042797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:34:30.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:30.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:30 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:30.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:30 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2357402261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:34:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fe00a3bb0 msgr2=0x7f8fe00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0058c0 tx=0x7f8fec005f90 comp rx=0 tx=0).stop 2024-04-13T03:34:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff8102980 msgr2=0x7f8ff810c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff8102980 0x7f8ff810c4f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009fd0 tx=0x7f8ff000ef10 comp rx=0 tx=0).stop 2024-04-13T03:34:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 shutdown_connections 2024-04-13T03:34:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff8102980 0x7f8ff810c4f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ff8101fc0 0x7f8ff810bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 --2- 172.21.15.16:0/2264308772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff8100dd0 0x7f8ff810d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:30.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.873+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 >> 172.21.15.16:0/2264308772 conn(0x7f8ff80fc360 msgr2=0x7f8ff8105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:30.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.874+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 shutdown_connections 2024-04-13T03:34:30.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:30.874+0000 7f8ffe200640 1 -- 172.21.15.16:0/2264308772 wait complete. 2024-04-13T03:34:30.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-13T03:34:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:30 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3878042797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:34:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:30 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:30 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2357402261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:34:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.128+0000 7fee2b200640 1 -- 172.21.15.16:0/4269842742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c102950 msgr2=0x7fee2c104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.128+0000 7fee2b200640 1 --2- 172.21.15.16:0/4269842742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c102950 0x7fee2c104d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009ca0 tx=0x7fee1c02c100 comp rx=0 tx=0).stop 2024-04-13T03:34:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.128+0000 7fee2b200640 1 -- 172.21.15.16:0/4269842742 shutdown_connections 2024-04-13T03:34:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.128+0000 7fee2b200640 1 --2- 172.21.15.16:0/4269842742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee2c107bb0 0x7fee2c10a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.128+0000 7fee2b200640 1 --2- 172.21.15.16:0/4269842742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee2c105280 0x7fee2c107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.128+0000 7fee2b200640 1 --2- 172.21.15.16:0/4269842742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c102950 0x7fee2c104d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.128+0000 7fee2b200640 1 -- 172.21.15.16:0/4269842742 >> 172.21.15.16:0/4269842742 conn(0x7fee2c0fc360 msgr2=0x7fee2c0fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.128+0000 7fee2b200640 1 -- 172.21.15.16:0/4269842742 shutdown_connections 2024-04-13T03:34:31.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.128+0000 7fee2b200640 1 -- 172.21.15.16:0/4269842742 wait complete. 2024-04-13T03:34:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.129+0000 7fee2b200640 1 Processor -- start 2024-04-13T03:34:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee2b200640 1 -- start start 2024-04-13T03:34:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee2b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee2c102950 0x7fee2c19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee2b200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c105280 0x7fee2c19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee2b200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee2c107bb0 0x7fee2c1a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee2b200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c10db40 con 0x7fee2c107bb0 2024-04-13T03:34:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee2b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c10d9c0 con 0x7fee2c102950 2024-04-13T03:34:31.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee2b200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c10dcc0 con 0x7fee2c105280 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee2a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee2c107bb0 0x7fee2c1a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee29400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c105280 0x7fee2c19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee29e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee2c102950 0x7fee2c19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee29400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c105280 0x7fee2c19a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53898/0 (socket says 172.21.15.16:53898) 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee2a800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee2c107bb0 0x7fee2c1a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38866/0 (socket says 172.21.15.16:38866) 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.130+0000 7fee29400640 1 -- 172.21.15.16:0/3798067178 learned_addr learned my addr 172.21.15.16:0/3798067178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee29400640 1 -- 172.21.15.16:0/3798067178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee2c102950 msgr2=0x7fee2c19a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee29400640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee2c102950 0x7fee2c19a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee29400640 1 -- 172.21.15.16:0/3798067178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee2c107bb0 msgr2=0x7fee2c1a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee29400640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee2c107bb0 0x7fee2c1a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee29400640 1 -- 172.21.15.16:0/3798067178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c1a1fd0 con 0x7fee2c105280 2024-04-13T03:34:31.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee29e00640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee2c102950 0x7fee2c19a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee29400640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c105280 0x7fee2c19a8f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fee1800e9a0 tx=0x7fee1800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee22a00640 1 -- 172.21.15.16:0/3798067178 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1800cd80 con 0x7fee2c105280 2024-04-13T03:34:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee22a00640 1 -- 172.21.15.16:0/3798067178 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee180035f0 con 0x7fee2c105280 2024-04-13T03:34:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2c1a22c0 con 0x7fee2c105280 2024-04-13T03:34:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.131+0000 7fee22a00640 1 -- 172.21.15.16:0/3798067178 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee18010670 con 0x7fee2c105280 2024-04-13T03:34:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.132+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2c1a2810 con 0x7fee2c105280 2024-04-13T03:34:31.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.133+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2c109700 con 0x7fee2c105280 2024-04-13T03:34:31.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.135+0000 7fee22a00640 1 -- 172.21.15.16:0/3798067178 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fee180037a0 con 0x7fee2c105280 2024-04-13T03:34:31.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.136+0000 7fee22a00640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee100a3bb0 0x7fee100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:31.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.137+0000 7fee22a00640 1 -- 172.21.15.16:0/3798067178 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fee180d0ab0 con 0x7fee2c105280 2024-04-13T03:34:31.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.137+0000 7fee29e00640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee100a3bb0 0x7fee100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:31.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.137+0000 7fee29e00640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee100a3bb0 0x7fee100a6070 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fee1c02c710 tx=0x7fee1c005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:31.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.141+0000 7fee22a00640 1 -- 172.21.15.16:0/3798067178 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee1809d060 con 0x7fee2c105280 2024-04-13T03:34:31.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.354+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fee2c062bc0 con 0x7fee2c105280 2024-04-13T03:34:31.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.355+0000 7fee22a00640 1 -- 172.21.15.16:0/3798067178 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fee1809c960 con 0x7fee2c105280 2024-04-13T03:34:31.355 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.357+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee100a3bb0 msgr2=0x7fee100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.357+0000 7fee2b200640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee100a3bb0 0x7fee100a6070 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fee1c02c710 tx=0x7fee1c005ef0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.357+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c105280 msgr2=0x7fee2c19a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.357+0000 7fee2b200640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c105280 0x7fee2c19a8f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fee1800e9a0 tx=0x7fee1800ee60 comp rx=0 tx=0).stop 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.358+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 shutdown_connections 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.358+0000 7fee2b200640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fee100a3bb0 0x7fee100a6070 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.358+0000 7fee2b200640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fee2c107bb0 0x7fee2c1a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.358+0000 7fee2b200640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee2c105280 0x7fee2c19a8f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.358+0000 7fee2b200640 1 --2- 172.21.15.16:0/3798067178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee2c102950 0x7fee2c19a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.358+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 >> 172.21.15.16:0/3798067178 conn(0x7fee2c0fc360 msgr2=0x7fee2c102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:31.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.358+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 shutdown_connections 2024-04-13T03:34:31.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.358+0000 7fee2b200640 1 -- 172.21.15.16:0/3798067178 wait complete. 2024-04-13T03:34:31.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-13T03:34:31.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.592+0000 7f45b6200640 1 -- 172.21.15.16:0/1790488752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0109d60 msgr2=0x7f45b010c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.592+0000 7f45b6200640 1 --2- 172.21.15.16:0/1790488752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0109d60 0x7f45b010c200 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009ca0 tx=0x7f45a802c180 comp rx=0 tx=0).stop 2024-04-13T03:34:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.592+0000 7f45b6200640 1 -- 172.21.15.16:0/1790488752 shutdown_connections 2024-04-13T03:34:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.592+0000 7f45b6200640 1 --2- 172.21.15.16:0/1790488752 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0109d60 0x7f45b010c200 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.592+0000 7f45b6200640 1 --2- 172.21.15.16:0/1790488752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b00749c0 0x7f45b0074e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.592+0000 7f45b6200640 1 --2- 172.21.15.16:0/1790488752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45b0075fa0 0x7f45b00743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.592+0000 7f45b6200640 1 -- 172.21.15.16:0/1790488752 >> 172.21.15.16:0/1790488752 conn(0x7f45b00fd460 msgr2=0x7f45b00ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.592+0000 7f45b6200640 1 -- 172.21.15.16:0/1790488752 shutdown_connections 2024-04-13T03:34:31.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.592+0000 7f45b6200640 1 -- 172.21.15.16:0/1790488752 wait complete. 2024-04-13T03:34:31.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.593+0000 7f45b6200640 1 Processor -- start 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.593+0000 7f45b6200640 1 -- start start 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45b6200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b00749c0 0x7f45b019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45b6200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0075fa0 0x7f45b019a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45b6200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45b0109d60 0x7f45b01a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45b6200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f45b010fd60 con 0x7f45b0075fa0 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45b6200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f45b010fbe0 con 0x7f45b0109d60 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45b6200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f45b010fee0 con 0x7f45b00749c0 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45afe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0075fa0 0x7f45b019a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45afe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0075fa0 0x7f45b019a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38894/0 (socket says 172.21.15.16:38894) 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45afe00640 1 -- 172.21.15.16:0/2163510569 learned_addr learned my addr 172.21.15.16:0/2163510569 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45b5800640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45b0109d60 0x7f45b01a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.594+0000 7f45b4e00640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b00749c0 0x7f45b019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45afe00640 1 -- 172.21.15.16:0/2163510569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b00749c0 msgr2=0x7f45b019a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45afe00640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b00749c0 0x7f45b019a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45afe00640 1 -- 172.21.15.16:0/2163510569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45b0109d60 msgr2=0x7f45b01a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45afe00640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45b0109d60 0x7f45b01a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45afe00640 1 -- 172.21.15.16:0/2163510569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b01a1f50 con 0x7f45b0075fa0 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45b5800640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45b0109d60 0x7f45b01a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45afe00640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0075fa0 0x7f45b019a870 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f459c00d950 tx=0x7f459c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45ad600640 1 -- 172.21.15.16:0/2163510569 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c017070 con 0x7f45b0075fa0 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b01a2240 con 0x7f45b0075fa0 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.595+0000 7f45ad600640 1 -- 172.21.15.16:0/2163510569 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f459c009c30 con 0x7f45b0075fa0 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.596+0000 7f45ad600640 1 -- 172.21.15.16:0/2163510569 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c020060 con 0x7f45b0075fa0 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.596+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b01a2790 con 0x7f45b0075fa0 2024-04-13T03:34:31.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.597+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b0108b20 con 0x7f45b0075fa0 2024-04-13T03:34:31.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.599+0000 7f45ad600640 1 -- 172.21.15.16:0/2163510569 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f459c0109d0 con 0x7f45b0075fa0 2024-04-13T03:34:31.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.600+0000 7f45ad600640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f45980a3bb0 0x7f45980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:31.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.600+0000 7f45ad600640 1 -- 172.21.15.16:0/2163510569 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f459c0d05f0 con 0x7f45b0075fa0 2024-04-13T03:34:31.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.601+0000 7f45b4e00640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f45980a3bb0 0x7f45980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:31.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.602+0000 7f45b4e00640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f45980a3bb0 0x7f45980a6070 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009bd0 tx=0x7f45a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:31.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.603+0000 7f45ad600640 1 -- 172.21.15.16:0/2163510569 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f459c009de0 con 0x7f45b0075fa0 2024-04-13T03:34:31.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.812+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f45b002d830 con 0x7f45b0075fa0 2024-04-13T03:34:31.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.813+0000 7f45ad600640 1 -- 172.21.15.16:0/2163510569 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f459c09c550 con 0x7f45b0075fa0 2024-04-13T03:34:31.813 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.815+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f45980a3bb0 msgr2=0x7f45980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.815+0000 7f45b6200640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f45980a3bb0 0x7f45980a6070 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009bd0 tx=0x7f45a4005f90 comp rx=0 tx=0).stop 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.815+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0075fa0 msgr2=0x7f45b019a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.815+0000 7f45b6200640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0075fa0 0x7f45b019a870 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f459c00d950 tx=0x7f459c00de10 comp rx=0 tx=0).stop 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.816+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 shutdown_connections 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.816+0000 7f45b6200640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f45980a3bb0 0x7f45980a6070 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.816+0000 7f45b6200640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45b0109d60 0x7f45b01a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.816+0000 7f45b6200640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45b0075fa0 0x7f45b019a870 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.816+0000 7f45b6200640 1 --2- 172.21.15.16:0/2163510569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45b00749c0 0x7f45b019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.816+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 >> 172.21.15.16:0/2163510569 conn(0x7f45b00fd460 msgr2=0x7f45b0105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.816+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 shutdown_connections 2024-04-13T03:34:31.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:31.816+0000 7f45b6200640 1 -- 172.21.15.16:0/2163510569 wait complete. 2024-04-13T03:34:31.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:31 smithi016 ceph-mon[31772]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:34:31.832 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:31 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2264308772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:34:31.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:31.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:31.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:31 smithi016 ceph-mon[31772]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:31.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:31 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3798067178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:34:31.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:34:31.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:31 smithi188 ceph-mon[36398]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:34:31.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:31 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2264308772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:34:31.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:31.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:31.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:31 smithi188 ceph-mon[36398]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:31.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:31 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3798067178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:34:31.973 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:31 smithi119 ceph-mon[35835]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-13T03:34:31.973 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:31 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2264308772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:34:31.973 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:31.973 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:31.973 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:31 smithi119 ceph-mon[35835]: from='mgr.14152 172.21.15.16:0/3807942434' entity='mgr.x' 2024-04-13T03:34:31.973 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:31 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3798067178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:34:32.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.047+0000 7f0e86000640 1 -- 172.21.15.16:0/1864025114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e80101e60 msgr2=0x7f0e80102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.047+0000 7f0e86000640 1 --2- 172.21.15.16:0/1864025114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e80101e60 0x7f0e80102280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009be0 tx=0x7f0e7002bef0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.047+0000 7f0e86000640 1 -- 172.21.15.16:0/1864025114 shutdown_connections 2024-04-13T03:34:32.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.047+0000 7f0e86000640 1 --2- 172.21.15.16:0/1864025114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80103970 0x7f0e8010a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.047+0000 7f0e86000640 1 --2- 172.21.15.16:0/1864025114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e80102fb0 0x7f0e80103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.047+0000 7f0e86000640 1 --2- 172.21.15.16:0/1864025114 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e80101e60 0x7f0e80102280 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.047+0000 7f0e86000640 1 -- 172.21.15.16:0/1864025114 >> 172.21.15.16:0/1864025114 conn(0x7f0e800fd3d0 msgr2=0x7f0e800ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:32.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.047+0000 7f0e86000640 1 -- 172.21.15.16:0/1864025114 shutdown_connections 2024-04-13T03:34:32.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.047+0000 7f0e86000640 1 -- 172.21.15.16:0/1864025114 wait complete. 2024-04-13T03:34:32.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.048+0000 7f0e86000640 1 Processor -- start 2024-04-13T03:34:32.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.049+0000 7f0e86000640 1 -- start start 2024-04-13T03:34:32.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.049+0000 7f0e86000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e80101e60 0x7f0e80075d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:32.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.049+0000 7f0e86000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102fb0 0x7f0e800743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:32.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.049+0000 7f0e86000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e80103970 0x7f0e800748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:32.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.049+0000 7f0e86000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8010db50 con 0x7f0e80101e60 2024-04-13T03:34:32.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.049+0000 7f0e86000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8010d9d0 con 0x7f0e80102fb0 2024-04-13T03:34:32.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.049+0000 7f0e86000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8010dcd0 con 0x7f0e80103970 2024-04-13T03:34:32.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e7fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102fb0 0x7f0e800743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:32.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e7fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102fb0 0x7f0e800743b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56016/0 (socket says 172.21.15.16:56016) 2024-04-13T03:34:32.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e7fe00640 1 -- 172.21.15.16:0/2764993259 learned_addr learned my addr 172.21.15.16:0/2764993259 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:32.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e85600640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e80103970 0x7f0e800748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:32.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e84c00640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e80101e60 0x7f0e80075d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:32.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e7fe00640 1 -- 172.21.15.16:0/2764993259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e80103970 msgr2=0x7f0e800748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e7fe00640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e80103970 0x7f0e800748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e7fe00640 1 -- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e80101e60 msgr2=0x7f0e80075d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e7fe00640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e80101e60 0x7f0e80075d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e7fe00640 1 -- 172.21.15.16:0/2764993259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e80075180 con 0x7f0e80102fb0 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.050+0000 7f0e84c00640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e80101e60 0x7f0e80075d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.051+0000 7f0e7fe00640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102fb0 0x7f0e800743b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800e9c0 tx=0x7f0e6800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.051+0000 7f0e7d600640 1 -- 172.21.15.16:0/2764993259 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6800ccc0 con 0x7f0e80102fb0 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.051+0000 7f0e7d600640 1 -- 172.21.15.16:0/2764993259 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e680035f0 con 0x7f0e80102fb0 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.051+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e801a2280 con 0x7f0e80102fb0 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.051+0000 7f0e7d600640 1 -- 172.21.15.16:0/2764993259 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6800ce60 con 0x7f0e80102fb0 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.051+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e801a27d0 con 0x7f0e80102fb0 2024-04-13T03:34:32.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.052+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e801091c0 con 0x7f0e80102fb0 2024-04-13T03:34:32.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.054+0000 7f0e7d600640 1 -- 172.21.15.16:0/2764993259 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0e680037a0 con 0x7f0e80102fb0 2024-04-13T03:34:32.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.056+0000 7f0e7d600640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0e640a3bb0 0x7f0e640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:32.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.056+0000 7f0e7d600640 1 -- 172.21.15.16:0/2764993259 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0e680d0eb0 con 0x7f0e80102fb0 2024-04-13T03:34:32.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.056+0000 7f0e84c00640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0e640a3bb0 0x7f0e640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:32.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.057+0000 7f0e84c00640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0e640a3bb0 0x7f0e640a6070 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0e70034fd0 tx=0x7f0e70000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:32.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.058+0000 7f0e7d600640 1 -- 172.21.15.16:0/2764993259 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e6809d460 con 0x7f0e80102fb0 2024-04-13T03:34:32.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.308+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e8002d5e0 con 0x7f0e80102fb0 2024-04-13T03:34:32.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.309+0000 7f0e7d600640 1 -- 172.21.15.16:0/2764993259 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11934 (secure 0 0 0) 0x7f0e6809cd60 con 0x7f0e80102fb0 2024-04-13T03:34:32.309 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:34:32.310 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:32:07.796132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24530,"mds_2":24523,"mds_3":24505,"mds_4":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24505,24500,24587,24523,24530]},"id":1}]} 2024-04-13T03:34:32.310 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 26 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0e640a3bb0 msgr2=0x7f0e640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0e640a3bb0 0x7f0e640a6070 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0e70034fd0 tx=0x7f0e70000f40 comp rx=0 tx=0).stop 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102fb0 msgr2=0x7f0e800743b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102fb0 0x7f0e800743b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800e9c0 tx=0x7f0e6800edc0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 shutdown_connections 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f0e640a3bb0 0x7f0e640a6070 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e80103970 0x7f0e800748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e80102fb0 0x7f0e800743b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 --2- 172.21.15.16:0/2764993259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0e80101e60 0x7f0e80075d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.312+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 >> 172.21.15.16:0/2764993259 conn(0x7f0e800fd3d0 msgr2=0x7f0e80106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.313+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 shutdown_connections 2024-04-13T03:34:32.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.313+0000 7f0e86000640 1 -- 172.21.15.16:0/2764993259 wait complete. 2024-04-13T03:34:32.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:34:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.555+0000 7f68de800640 1 -- 172.21.15.16:0/3910778046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68d8109630 msgr2=0x7f68d8109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.555+0000 7f68de800640 1 --2- 172.21.15.16:0/3910778046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68d8109630 0x7f68d8109ab0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009860 tx=0x7f68c402bb70 comp rx=0 tx=0).stop 2024-04-13T03:34:32.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.555+0000 7f68de800640 1 -- 172.21.15.16:0/3910778046 shutdown_connections 2024-04-13T03:34:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.555+0000 7f68de800640 1 --2- 172.21.15.16:0/3910778046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d8109ff0 0x7f68d8110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.555+0000 7f68de800640 1 --2- 172.21.15.16:0/3910778046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68d8109630 0x7f68d8109ab0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.555+0000 7f68de800640 1 --2- 172.21.15.16:0/3910778046 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d8108440 0x7f68d8108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.555+0000 7f68de800640 1 -- 172.21.15.16:0/3910778046 >> 172.21.15.16:0/3910778046 conn(0x7f68d8077400 msgr2=0x7f68d8079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.555+0000 7f68de800640 1 -- 172.21.15.16:0/3910778046 shutdown_connections 2024-04-13T03:34:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.555+0000 7f68de800640 1 -- 172.21.15.16:0/3910778046 wait complete. 2024-04-13T03:34:32.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.556+0000 7f68de800640 1 Processor -- start 2024-04-13T03:34:32.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.556+0000 7f68de800640 1 -- start start 2024-04-13T03:34:32.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.556+0000 7f68de800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d8108440 0x7f68d819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:32.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68de800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68d8109630 0x7f68d819ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:32.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68de800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d8109ff0 0x7f68d81a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:32.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68de800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f68d81140d0 con 0x7f68d8109ff0 2024-04-13T03:34:32.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68de800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f68d8113f50 con 0x7f68d8109630 2024-04-13T03:34:32.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68de800640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f68d8114250 con 0x7f68d8108440 2024-04-13T03:34:32.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dd400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d8108440 0x7f68d819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dd400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d8108440 0x7f68d819e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53946/0 (socket says 172.21.15.16:53946) 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dd400640 1 -- 172.21.15.16:0/2607638959 learned_addr learned my addr 172.21.15.16:0/2607638959 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dde00640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d8109ff0 0x7f68d81a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dca00640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68d8109630 0x7f68d819ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dd400640 1 -- 172.21.15.16:0/2607638959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68d8109630 msgr2=0x7f68d819ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dd400640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68d8109630 0x7f68d819ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dd400640 1 -- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d8109ff0 msgr2=0x7f68d81a5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dd400640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d8109ff0 0x7f68d81a5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.557+0000 7f68dd400640 1 -- 172.21.15.16:0/2607638959 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d81a6350 con 0x7f68d8108440 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.558+0000 7f68dd400640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d8108440 0x7f68d819e780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00e9c0 tx=0x7f68cc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.558+0000 7f68d6000640 1 -- 172.21.15.16:0/2607638959 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc00cd80 con 0x7f68d8108440 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.558+0000 7f68d6000640 1 -- 172.21.15.16:0/2607638959 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68cc0035f0 con 0x7f68d8108440 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.558+0000 7f68d6000640 1 -- 172.21.15.16:0/2607638959 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68cc010690 con 0x7f68d8108440 2024-04-13T03:34:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.558+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d81a6640 con 0x7f68d8108440 2024-04-13T03:34:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.558+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d81a6b10 con 0x7f68d8108440 2024-04-13T03:34:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.559+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d810f760 con 0x7f68d8108440 2024-04-13T03:34:32.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.562+0000 7f68d6000640 1 -- 172.21.15.16:0/2607638959 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f68cc0037a0 con 0x7f68d8108440 2024-04-13T03:34:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.563+0000 7f68d6000640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f68c00a3bb0 0x7f68c00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.563+0000 7f68d6000640 1 -- 172.21.15.16:0/2607638959 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f68cc0d0c40 con 0x7f68d8108440 2024-04-13T03:34:32.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.563+0000 7f68dca00640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f68c00a3bb0 0x7f68c00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:32.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.564+0000 7f68dca00640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f68c00a3bb0 0x7f68c00a6070 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009830 tx=0x7f68c4034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.568+0000 7f68d6000640 1 -- 172.21.15.16:0/2607638959 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f68cc09d1f0 con 0x7f68d8108440 2024-04-13T03:34:32.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.784+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68d8062bc0 con 0x7f68d8108440 2024-04-13T03:34:32.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.787+0000 7f68d6000640 1 -- 172.21.15.16:0/2607638959 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24874 (secure 0 0 0) 0x7f68cc09caf0 con 0x7f68d8108440 2024-04-13T03:34:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:34:32.788 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":94,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:33:42.153326+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:34:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.789+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f68c00a3bb0 msgr2=0x7f68c00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.789+0000 7f68de800640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f68c00a3bb0 0x7f68c00a6070 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009830 tx=0x7f68c4034ad0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.789+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d8108440 msgr2=0x7f68d819e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.789+0000 7f68de800640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d8108440 0x7f68d819e780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f68cc00e9c0 tx=0x7f68cc00ee80 comp rx=0 tx=0).stop 2024-04-13T03:34:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.790+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 shutdown_connections 2024-04-13T03:34:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.790+0000 7f68de800640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f68c00a3bb0 0x7f68c00a6070 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.790+0000 7f68de800640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f68d8109ff0 0x7f68d81a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.790+0000 7f68de800640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68d8109630 0x7f68d819ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.790+0000 7f68de800640 1 --2- 172.21.15.16:0/2607638959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68d8108440 0x7f68d819e780 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.790+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 >> 172.21.15.16:0/2607638959 conn(0x7f68d8077400 msgr2=0x7f68d810c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.790+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 shutdown_connections 2024-04-13T03:34:32.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:32.790+0000 7f68de800640 1 -- 172.21.15.16:0/2607638959 wait complete. 2024-04-13T03:34:32.811 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-13T03:31:37.109492+0000', 'modified': '2024-04-13T03:32:07.796132+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24587, 'mds_1': 24530, 'mds_2': 24523, 'mds_3': 24505, 'mds_4': 24500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24433': {'gid': 24433, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6833/3805092939', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6832', 'nonce': 3805092939}, {'type': 'v1', 'addr': '172.21.15.188:6833', 'nonce': 3805092939}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6833/3221738952', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6832', 'nonce': 3221738952}, {'type': 'v1', 'addr': '172.21.15.119:6833', 'nonce': 3221738952}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6835/1177424525', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6834', 'nonce': 1177424525}, {'type': 'v1', 'addr': '172.21.15.16:6835', 'nonce': 1177424525}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.16:6837/2782624805', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6836', 'nonce': 2782624805}, {'type': 'v1', 'addr': '172.21.15.16:6837', 'nonce': 2782624805}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6835/566226134', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6834', 'nonce': 566226134}, {'type': 'v1', 'addr': '172.21.15.188:6835', 'nonce': 566226134}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'e', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.119:6835/4186042513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 4186042513}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 4186042513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.188:6837/284865646', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 284865646}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 284865646}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.188:6839/57242391', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 57242391}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 57242391}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.119:6837/477256695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6836', 'nonce': 477256695}, {'type': 'v1', 'addr': '172.21.15.119:6837', 'nonce': 477256695}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/3722020849', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 3722020849}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 3722020849}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24587, 'qdb_cluster': [24505, 24500, 24587, 24523, 24530]} 2024-04-13T03:34:32.811 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-13T03:34:32.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-13T03:34:32.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:32 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2163510569' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:34:32.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:32 smithi188 ceph-mon[36398]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:32.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:32 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2764993259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:34:32.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:32 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2163510569' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:34:32.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:32 smithi016 ceph-mon[31772]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:32.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:32 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2764993259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:34:32.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:32 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2163510569' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:34:32.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:32 smithi119 ceph-mon[35835]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-13T03:34:32.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:32 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2764993259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:34:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.048+0000 7fd4e1a00640 1 -- 172.21.15.16:0/4223669430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4dc105dd0 msgr2=0x7fd4dc10c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.048+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/4223669430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4dc105dd0 0x7fd4dc10c660 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009900 tx=0x7fd4d402bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.049+0000 7fd4e1a00640 1 -- 172.21.15.16:0/4223669430 shutdown_connections 2024-04-13T03:34:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.049+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/4223669430 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4dc105dd0 0x7fd4dc10c660 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.049+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/4223669430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4dc105410 0x7fd4dc105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.049+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/4223669430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4dc104220 0x7fd4dc104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.049+0000 7fd4e1a00640 1 -- 172.21.15.16:0/4223669430 >> 172.21.15.16:0/4223669430 conn(0x7fd4dc0ff7b0 msgr2=0x7fd4dc101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.049+0000 7fd4e1a00640 1 -- 172.21.15.16:0/4223669430 shutdown_connections 2024-04-13T03:34:33.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.049+0000 7fd4e1a00640 1 -- 172.21.15.16:0/4223669430 wait complete. 2024-04-13T03:34:33.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.050+0000 7fd4e1a00640 1 Processor -- start 2024-04-13T03:34:33.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.050+0000 7fd4e1a00640 1 -- start start 2024-04-13T03:34:33.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.050+0000 7fd4e1a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4dc104220 0x7fd4dc19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.050+0000 7fd4e1a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4dc105410 0x7fd4dc19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.050+0000 7fd4e1a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4dc105dd0 0x7fd4dc1a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.050+0000 7fd4e1a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd4dc10fe80 con 0x7fd4dc104220 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.050+0000 7fd4e1a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd4dc10fd00 con 0x7fd4dc105dd0 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.050+0000 7fd4e1a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd4dc110000 con 0x7fd4dc105410 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4db400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4dc105dd0 0x7fd4dc1a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4db400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4dc105dd0 0x7fd4dc1a1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56050/0 (socket says 172.21.15.16:56050) 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4db400640 1 -- 172.21.15.16:0/501751779 learned_addr learned my addr 172.21.15.16:0/501751779 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4daa00640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4dc104220 0x7fd4dc19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4da000640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4dc105410 0x7fd4dc19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4db400640 1 -- 172.21.15.16:0/501751779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4dc105410 msgr2=0x7fd4dc19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4db400640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4dc105410 0x7fd4dc19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4db400640 1 -- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4dc104220 msgr2=0x7fd4dc19a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4db400640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4dc104220 0x7fd4dc19a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4db400640 1 -- 172.21.15.16:0/501751779 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4dc1a2150 con 0x7fd4dc105dd0 2024-04-13T03:34:33.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4da000640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4dc105410 0x7fd4dc19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:33.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4daa00640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4dc104220 0x7fd4dc19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:33.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.051+0000 7fd4db400640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4dc105dd0 0x7fd4dc1a1ae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009230 tx=0x7fd4d402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.052+0000 7fd4d3400640 1 -- 172.21.15.16:0/501751779 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4041070 con 0x7fd4dc105dd0 2024-04-13T03:34:33.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.052+0000 7fd4d3400640 1 -- 172.21.15.16:0/501751779 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4d40054e0 con 0x7fd4dc105dd0 2024-04-13T03:34:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.052+0000 7fd4d3400640 1 -- 172.21.15.16:0/501751779 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4003d80 con 0x7fd4dc105dd0 2024-04-13T03:34:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.052+0000 7fd4e1a00640 1 -- 172.21.15.16:0/501751779 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4dc1a23e0 con 0x7fd4dc105dd0 2024-04-13T03:34:33.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.052+0000 7fd4e1a00640 1 -- 172.21.15.16:0/501751779 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4dc1a2880 con 0x7fd4dc105dd0 2024-04-13T03:34:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.055+0000 7fd4d3400640 1 -- 172.21.15.16:0/501751779 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd4d4044070 con 0x7fd4dc105dd0 2024-04-13T03:34:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.055+0000 7fd4e4983740 1 Processor -- start 2024-04-13T03:34:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4e4983740 1 -- start start 2024-04-13T03:34:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 0x55f28ef40a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef410d0 0x55f28ef43560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4e4983740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f28eea1990 con 0x55f28ef3b570 2024-04-13T03:34:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4e4983740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f28ee48530 con 0x55f28ef410d0 2024-04-13T03:34:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4e4983740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f28ee9d950 con 0x55f28ef3bf60 2024-04-13T03:34:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4da000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4da000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38946/0 (socket says 172.21.15.16:38946) 2024-04-13T03:34:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4da000640 1 -- 172.21.15.16:0/50987105 learned_addr learned my addr 172.21.15.16:0/50987105 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4daa00640 1 --2- 172.21.15.16:0/50987105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 0x55f28ef40a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.056+0000 7fd4da000640 1 --2- 172.21.15.16:0/50987105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef410d0 0x55f28ef43560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4daa00640 1 -- 172.21.15.16:0/50987105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef410d0 msgr2=0x55f28ef43560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4d3400640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd4c00a3c30 0x7fd4c00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4daa00640 1 --2- 172.21.15.16:0/50987105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef410d0 0x55f28ef43560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4daa00640 1 -- 172.21.15.16:0/50987105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 msgr2=0x55f28ef3b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4daa00640 1 --2- 172.21.15.16:0/50987105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4daa00640 1 -- 172.21.15.16:0/50987105 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f28ebb42d0 con 0x55f28ef3bf60 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4d3400640 1 -- 172.21.15.16:0/501751779 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd4d40f23f0 con 0x7fd4dc105dd0 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4daa00640 1 --2- 172.21.15.16:0/50987105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 0x55f28ef40a60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc005fd0 tx=0x7fd4cc009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27d1cd4a5eb2e188 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4db400640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd4c00a3c30 0x7fd4c00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.057+0000 7fd4c7e00640 1 -- 172.21.15.16:0/50987105 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc004d20 con 0x55f28ef3bf60 2024-04-13T03:34:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4c7e00640 1 -- 172.21.15.16:0/50987105 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4cc015070 con 0x55f28ef3bf60 2024-04-13T03:34:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4c7e00640 1 -- 172.21.15.16:0/50987105 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc011500 con 0x55f28ef3bf60 2024-04-13T03:34:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4e4983740 1 -- 172.21.15.16:0/50987105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 msgr2=0x55f28ef40a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4db400640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd4c00a3c30 0x7fd4c00a60f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009430 tx=0x7fd4d402bc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4e4983740 1 --2- 172.21.15.16:0/50987105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 0x55f28ef40a60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc005fd0 tx=0x7fd4cc009f90 comp rx=0 tx=0).stop 2024-04-13T03:34:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4e4983740 1 -- 172.21.15.16:0/50987105 shutdown_connections 2024-04-13T03:34:33.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4e4983740 1 --2- 172.21.15.16:0/50987105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef410d0 0x55f28ef43560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4e4983740 1 --2- 172.21.15.16:0/50987105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 0x55f28ef40a60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4e4983740 1 --2- 172.21.15.16:0/50987105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4e4983740 1 -- 172.21.15.16:0/50987105 >> 172.21.15.16:0/50987105 conn(0x55f28ef36ae0 msgr2=0x55f28ef36ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4e4983740 1 -- 172.21.15.16:0/50987105 shutdown_connections 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.058+0000 7fd4e4983740 1 -- 172.21.15.16:0/50987105 wait complete. 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.059+0000 7fd4e4983740 1 -- start start 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.059+0000 7fd4e4983740 1 Processor -- start 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3b570 0x55f28ef45af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3bf60 0x55f28ef46030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef410d0 0x55f28ef4d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4e4983740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f28ef44080 con 0x55f28ef3bf60 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4e4983740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f28ee5b8f0 con 0x55f28ef3b570 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4e4983740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f28ef44200 con 0x55f28ef410d0 2024-04-13T03:34:33.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4e4983740 10 client.0 _ll_register_callbacks cb 0x55f28ee5b2e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4daa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3bf60 0x55f28ef46030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4da000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3b570 0x55f28ef45af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4daa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3bf60 0x55f28ef46030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38952/0 (socket says 172.21.15.16:38952) 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4da000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3b570 0x55f28ef45af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56064/0 (socket says 172.21.15.16:56064) 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4daa00640 1 -- 172.21.15.16:0/2315949020 learned_addr learned my addr 172.21.15.16:0/2315949020 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4da000640 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef410d0 0x55f28ef4d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4daa00640 1 -- 172.21.15.16:0/2315949020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef410d0 msgr2=0x55f28ef4d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4daa00640 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef410d0 0x55f28ef4d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4daa00640 1 -- 172.21.15.16:0/2315949020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3b570 msgr2=0x55f28ef45af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4daa00640 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3b570 0x55f28ef45af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.060+0000 7fd4daa00640 1 -- 172.21.15.16:0/2315949020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f28ec7e6b0 con 0x55f28ef3bf60 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.061+0000 7fd4daa00640 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3bf60 0x55f28ef46030 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc018000 tx=0x7fd4cc003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8793fe8bec0b8f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.061+0000 7fd4c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.061+0000 7fd4c4c00640 1 -- 172.21.15.16:0/2315949020 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc004a40 con 0x55f28ef3bf60 2024-04-13T03:34:33.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.061+0000 7fd4e4983740 1 -- 172.21.15.16:0/2315949020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4cc015070 con 0x55f28ef3bf60 2024-04-13T03:34:33.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.061+0000 7fd4c4c00640 1 -- 172.21.15.16:0/2315949020 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd4cc004be0 con 0x55f28ef3bf60 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.061+0000 7fd4c4c00640 1 -- 172.21.15.16:0/2315949020 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd4cc02f050 con 0x55f28ef3bf60 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.061+0000 7fd4e4983740 10 client.15264 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.061+0000 7fd4e4983740 1 -- 172.21.15.16:0/2315949020 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f28ef50430 con 0x55f28ef3bf60 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.062+0000 7fd4c4c00640 1 -- 172.21.15.16:0/2315949020 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4cc015070 con 0x55f28ef3bf60 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.062+0000 7fd4c4c00640 1 -- 172.21.15.16:0/2315949020 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fd4cc011c60 con 0x55f28ef3bf60 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.062+0000 7fd4e4983740 10 client.15264 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.062+0000 7fd4e4983740 10 client.15264 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.062+0000 7fd4e4983740 20 client.15264 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.062+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x55f28ef52680 0x55f28ef53af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.062+0000 7fd4e4983740 4 client.15264 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.062+0000 7fd4e4983740 1 -- 172.21.15.16:0/2315949020 --> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f28ee47b40 con 0x55f28ef52680 2024-04-13T03:34:33.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.063+0000 7fd4da000640 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x55f28ef52680 0x55f28ef53af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.063+0000 7fd4da000640 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x55f28ef52680 0x55f28ef53af0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e85974ecf0b4d3fa server_cookie=aa7c91bd8698d558 in_seq=0 out_seq=0 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.063+0000 7fd4c4c00640 10 client.15264 ms_handle_connect on v2:172.21.15.119:6834/4186042513 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4c4c00640 1 -- 172.21.15.16:0/2315949020 <== mds.4 v2:172.21.15.119:6834/4186042513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f28ee47b40 con 0x55f28ef52680 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4c4c00640 10 client.15264 handle_command_reply: tid=0 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 client.15264 shutdown 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 -- 172.21.15.16:0/2315949020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3bf60 msgr2=0x55f28ef46030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3bf60 0x55f28ef46030 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc018000 tx=0x7fd4cc003fd0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 -- 172.21.15.16:0/2315949020 shutdown_connections 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x55f28ef52680 0x55f28ef53af0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef410d0 0x55f28ef4d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3bf60 0x55f28ef46030 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2315949020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3b570 0x55f28ef45af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 -- 172.21.15.16:0/2315949020 >> 172.21.15.16:0/2315949020 conn(0x55f28ef36ae0 msgr2=0x55f28ef35d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4c4c00640 0 client.15264 ms_handle_reset on v2:172.21.15.119:6834/4186042513 2024-04-13T03:34:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.064+0000 7fd4e4983740 1 -- 172.21.15.16:0/2315949020 shutdown_connections 2024-04-13T03:34:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.065+0000 7fd4e4983740 1 -- 172.21.15.16:0/2315949020 wait complete. 2024-04-13T03:34:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.065+0000 7fd4e4983740 20 client.15264 trim_cache size 0 max 16384 2024-04-13T03:34:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.084+0000 7fd4e4983740 1 Processor -- start 2024-04-13T03:34:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4e4983740 1 -- start start 2024-04-13T03:34:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef36ae0 0x55f28ef51840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef36ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3bf60 0x55f28ef45700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4e4983740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f28ef51e30 con 0x55f28ef3b570 2024-04-13T03:34:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4e4983740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f28ef52500 con 0x55f28ef3bf60 2024-04-13T03:34:33.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4e4983740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f28ef52bd0 con 0x55f28ef36ae0 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4daa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef36ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4da000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef36ae0 0x55f28ef51840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4da000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef36ae0 0x55f28ef51840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:53992/0 (socket says 172.21.15.16:53992) 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4da000640 1 -- 172.21.15.16:0/2528781149 learned_addr learned my addr 172.21.15.16:0/2528781149 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.085+0000 7fd4da000640 1 --2- 172.21.15.16:0/2528781149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3bf60 0x55f28ef45700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.086+0000 7fd4daa00640 1 -- 172.21.15.16:0/2528781149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef36ae0 msgr2=0x55f28ef51840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.086+0000 7fd4daa00640 1 --2- 172.21.15.16:0/2528781149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef36ae0 0x55f28ef51840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.086+0000 7fd4daa00640 1 -- 172.21.15.16:0/2528781149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3bf60 msgr2=0x55f28ef45700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.086+0000 7fd4daa00640 1 --2- 172.21.15.16:0/2528781149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3bf60 0x55f28ef45700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.086+0000 7fd4daa00640 1 -- 172.21.15.16:0/2528781149 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f28ef3c340 con 0x55f28ef3b570 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.086+0000 7fd4da000640 1 --2- 172.21.15.16:0/2528781149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef36ae0 0x55f28ef51840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.086+0000 7fd4daa00640 1 --2- 172.21.15.16:0/2528781149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef36ec0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc000f80 tx=0x7fd4cc009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40312642b4ca5f2a server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4c7400640 1 -- 172.21.15.16:0/2528781149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc005310 con 0x55f28ef3b570 2024-04-13T03:34:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4c7400640 1 -- 172.21.15.16:0/2528781149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4cc015040 con 0x55f28ef3b570 2024-04-13T03:34:33.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4c7400640 1 -- 172.21.15.16:0/2528781149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc004280 con 0x55f28ef3b570 2024-04-13T03:34:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4e4983740 1 -- 172.21.15.16:0/2528781149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 msgr2=0x55f28ef36ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2528781149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef36ec0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc000f80 tx=0x7fd4cc009fc0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4e4983740 1 -- 172.21.15.16:0/2528781149 shutdown_connections 2024-04-13T03:34:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2528781149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef3bf60 0x55f28ef45700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2528781149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef36ec0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4e4983740 1 --2- 172.21.15.16:0/2528781149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef36ae0 0x55f28ef51840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4e4983740 1 -- 172.21.15.16:0/2528781149 >> 172.21.15.16:0/2528781149 conn(0x55f28ef3cd40 msgr2=0x55f28ef31ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4e4983740 1 -- 172.21.15.16:0/2528781149 shutdown_connections 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.087+0000 7fd4e4983740 1 -- 172.21.15.16:0/2528781149 wait complete. 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.088+0000 7fd4e4983740 1 -- start start 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.088+0000 7fd4e4983740 1 Processor -- start 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.088+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef36ae0 0x55f28ef3f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.088+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.088+0000 7fd4e4983740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 0x55f28ef4b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.088+0000 7fd4e4983740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55f28ef53970 con 0x55f28ef3b570 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.088+0000 7fd4e4983740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f28ef532a0 con 0x55f28ef36ae0 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.088+0000 7fd4e4983740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55f28ef54040 con 0x55f28ef3bf60 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4e4983740 10 client.0 _ll_register_callbacks cb 0x7fd4cc009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef36ae0 0x55f28ef3f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4daa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef36ae0 0x55f28ef3f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56090/0 (socket says 172.21.15.16:56090) 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 -- 172.21.15.16:0/3666112349 learned_addr learned my addr 172.21.15.16:0/3666112349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 0x55f28ef4b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 -- 172.21.15.16:0/3666112349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 msgr2=0x55f28ef4b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 0x55f28ef4b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 -- 172.21.15.16:0/3666112349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 msgr2=0x55f28ef3fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 -- 172.21.15.16:0/3666112349 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f28ef4bcf0 con 0x55f28ef36ae0 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4daa00640 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4da000640 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef36ae0 0x55f28ef3f630 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55f28ef3c7c0 tx=0x7fd4c802b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5b17da4389c364 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.089+0000 7fd4c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.090+0000 7fd4c4c00640 1 -- 172.21.15.16:0/3666112349 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8027e60 con 0x55f28ef36ae0 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.090+0000 7fd4e4983740 1 -- 172.21.15.16:0/3666112349 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4cc015040 con 0x55f28ef36ae0 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.090+0000 7fd4c4c00640 1 -- 172.21.15.16:0/3666112349 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd4c80132b0 con 0x55f28ef36ae0 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.090+0000 7fd4c4c00640 1 -- 172.21.15.16:0/3666112349 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd4c8037050 con 0x55f28ef36ae0 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.090+0000 7fd4e4983740 10 client.25376 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.090+0000 7fd4e4983740 1 -- 172.21.15.16:0/3666112349 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f28efc13e0 con 0x55f28ef36ae0 2024-04-13T03:34:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.090+0000 7fd4c4c00640 1 -- 172.21.15.16:0/3666112349 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4cc015040 con 0x55f28ef36ae0 2024-04-13T03:34:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.090+0000 7fd4c4c00640 1 -- 172.21.15.16:0/3666112349 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7fd4c8032030 con 0x55f28ef36ae0 2024-04-13T03:34:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.090+0000 7fd4e4983740 10 client.25376 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.091+0000 7fd4e4983740 10 client.25376 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-13T03:34:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.091+0000 7fd4e4983740 20 client.25376 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.091+0000 7fd4e4983740 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x55f28ef410d0 0x55f28efc4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.091+0000 7fd4e4983740 4 client.25376 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:34:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.091+0000 7fd4e4983740 1 -- 172.21.15.16:0/3666112349 --> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] -- command(tid 0: {"prefix": "status"}) -- 0x55f28eeaf3b0 con 0x55f28ef410d0 2024-04-13T03:34:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.091+0000 7fd4daa00640 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x55f28ef410d0 0x55f28efc4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4daa00640 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x55f28ef410d0 0x55f28efc4c70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b2a56ae3498614b3 server_cookie=6513c4d65571d63 in_seq=0 out_seq=0 2024-04-13T03:34:33.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4c4c00640 10 client.25376 ms_handle_connect on v2:172.21.15.119:6834/4186042513 2024-04-13T03:34:33.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4c4c00640 1 -- 172.21.15.16:0/3666112349 <== mds.4 v2:172.21.15.119:6834/4186042513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55f28eeaf3b0 con 0x55f28ef410d0 2024-04-13T03:34:33.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4c4c00640 10 client.25376 handle_command_reply: tid=0 2024-04-13T03:34:33.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 client.25376 shutdown 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 -- 172.21.15.16:0/3666112349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef36ae0 msgr2=0x55f28ef3f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef36ae0 0x55f28ef3f630 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55f28ef3c7c0 tx=0x7fd4c802b040 comp rx=0 tx=0).stop 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 -- 172.21.15.16:0/3666112349 shutdown_connections 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.119:6834/4186042513,v1:172.21.15.119:6835/4186042513] conn(0x55f28ef410d0 0x55f28efc4c70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55f28ef3bf60 0x55f28ef4b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55f28ef3b570 0x55f28ef3fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 --2- 172.21.15.16:0/3666112349 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f28ef36ae0 0x55f28ef3f630 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 -- 172.21.15.16:0/3666112349 >> 172.21.15.16:0/3666112349 conn(0x55f28ef3cd40 msgr2=0x55f28ef36ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4c4c00640 0 client.25376 ms_handle_reset on v2:172.21.15.119:6834/4186042513 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 -- 172.21.15.16:0/3666112349 shutdown_connections 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.092+0000 7fd4e4983740 1 -- 172.21.15.16:0/3666112349 wait complete. 2024-04-13T03:34:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e4983740 20 client.25376 trim_cache size 0 max 16384 2024-04-13T03:34:33.097 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:34:33.097 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:34:33.097 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 4, 2024-04-13T03:34:33.097 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24500, 2024-04-13T03:34:33.097 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:34:33.097 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:34:33.097 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:34:33.097 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 146.02108665399999, 2024-04-13T03:34:33.097 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:34:33.098 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:34:33.098 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:34:33.098 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 205.00908386399999 2024-04-13T03:34:33.098 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:34:33.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 -- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd4c00a3c30 msgr2=0x7fd4c00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd4c00a3c30 0x7fd4c00a60f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009430 tx=0x7fd4d402bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:33.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 -- 172.21.15.16:0/501751779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4dc105dd0 msgr2=0x7fd4dc1a1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4dc105dd0 0x7fd4dc1a1ae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009230 tx=0x7fd4d402c650 comp rx=0 tx=0).stop 2024-04-13T03:34:33.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 -- 172.21.15.16:0/501751779 shutdown_connections 2024-04-13T03:34:33.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fd4c00a3c30 0x7fd4c00a60f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4dc105dd0 0x7fd4dc1a1ae0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4dc105410 0x7fd4dc19aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 --2- 172.21.15.16:0/501751779 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd4dc104220 0x7fd4dc19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.093+0000 7fd4e1a00640 1 -- 172.21.15.16:0/501751779 >> 172.21.15.16:0/501751779 conn(0x7fd4dc0ff7b0 msgr2=0x7fd4dc1086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.094+0000 7fd4e1a00640 1 -- 172.21.15.16:0/501751779 shutdown_connections 2024-04-13T03:34:33.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.094+0000 7fd4e1a00640 1 -- 172.21.15.16:0/501751779 wait complete. 2024-04-13T03:34:33.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-13T03:34:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.330+0000 7f40d8200640 1 -- 172.21.15.16:0/3081733561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0105dd0 msgr2=0x7f40d010c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.330+0000 7f40d8200640 1 --2- 172.21.15.16:0/3081733561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0105dd0 0x7f40d010c660 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009900 tx=0x7f40c802bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.330+0000 7f40d8200640 1 -- 172.21.15.16:0/3081733561 shutdown_connections 2024-04-13T03:34:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.331+0000 7f40d8200640 1 --2- 172.21.15.16:0/3081733561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0105dd0 0x7f40d010c660 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.331+0000 7f40d8200640 1 --2- 172.21.15.16:0/3081733561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d0105410 0x7f40d0105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.331+0000 7f40d8200640 1 --2- 172.21.15.16:0/3081733561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0104220 0x7f40d0104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.331+0000 7f40d8200640 1 -- 172.21.15.16:0/3081733561 >> 172.21.15.16:0/3081733561 conn(0x7f40d00ff7b0 msgr2=0x7f40d0101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.331+0000 7f40d8200640 1 -- 172.21.15.16:0/3081733561 shutdown_connections 2024-04-13T03:34:33.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.331+0000 7f40d8200640 1 -- 172.21.15.16:0/3081733561 wait complete. 2024-04-13T03:34:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d8200640 1 Processor -- start 2024-04-13T03:34:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d8200640 1 -- start start 2024-04-13T03:34:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d8200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0104220 0x7f40d019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d8200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0105410 0x7f40d019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d8200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d0105dd0 0x7f40d01a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d8200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f40d010ff00 con 0x7f40d0105410 2024-04-13T03:34:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d8200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f40d010fd80 con 0x7f40d0104220 2024-04-13T03:34:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d8200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f40d0110080 con 0x7f40d0105dd0 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d5800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0104220 0x7f40d019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0105410 0x7f40d019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d5800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0104220 0x7f40d019a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56112/0 (socket says 172.21.15.16:56112) 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.332+0000 7f40d5800640 1 -- 172.21.15.16:0/2601436393 learned_addr learned my addr 172.21.15.16:0/2601436393 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40d6200640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d0105dd0 0x7f40d01a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40d5800640 1 -- 172.21.15.16:0/2601436393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d0105dd0 msgr2=0x7f40d01a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40d5800640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d0105dd0 0x7f40d01a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40d5800640 1 -- 172.21.15.16:0/2601436393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0105410 msgr2=0x7f40d019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40d5800640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0105410 0x7f40d019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40d5800640 1 -- 172.21.15.16:0/2601436393 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d01a20e0 con 0x7f40d0104220 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40d5800640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0104220 0x7f40d019a4c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f40c400ebc0 tx=0x7f40c400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40ce000640 1 -- 172.21.15.16:0/2601436393 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4003b70 con 0x7f40d0104220 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40d8200640 1 -- 172.21.15.16:0/2601436393 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d01a23d0 con 0x7f40d0104220 2024-04-13T03:34:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40ce000640 1 -- 172.21.15.16:0/2601436393 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c4003d10 con 0x7f40d0104220 2024-04-13T03:34:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.333+0000 7f40ce000640 1 -- 172.21.15.16:0/2601436393 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4019980 con 0x7f40d0104220 2024-04-13T03:34:33.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.334+0000 7f40d8200640 1 -- 172.21.15.16:0/2601436393 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d01a2920 con 0x7f40d0104220 2024-04-13T03:34:33.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40db1f4740 1 Processor -- start 2024-04-13T03:34:33.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40db1f4740 1 -- start start 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40ce000640 1 -- 172.21.15.16:0/2601436393 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f40c4019b20 con 0x7f40d0104220 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 0x559285920990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x559285925a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285928560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40db1f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559285886990 con 0x559285920f60 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40db1f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55928582d530 con 0x5592859260d0 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40db1f4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559285882950 con 0x559285920570 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40d4e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285928560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.337+0000 7f40d5800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x559285925a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d4e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285928560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56124/0 (socket says 172.21.15.16:56124) 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d4e00640 1 -- 172.21.15.16:0/1599799734 learned_addr learned my addr 172.21.15.16:0/1599799734 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d6200640 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 0x559285920990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d4e00640 1 -- 172.21.15.16:0/1599799734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 msgr2=0x559285920990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d4e00640 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 0x559285920990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d4e00640 1 -- 172.21.15.16:0/1599799734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 msgr2=0x559285925a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d4e00640 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x559285925a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d4e00640 1 -- 172.21.15.16:0/1599799734 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592855992d0 con 0x5592859260d0 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d5800640 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x559285925a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d6200640 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 0x559285920990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40d4e00640 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285928560 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009ca0 tx=0x7f40bc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14ee006456ffe1ac server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40c2a00640 1 -- 172.21.15.16:0/1599799734 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc01e070 con 0x5592859260d0 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40c2a00640 1 -- 172.21.15.16:0/1599799734 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40bc019b10 con 0x5592859260d0 2024-04-13T03:34:33.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.338+0000 7f40c2a00640 1 -- 172.21.15.16:0/1599799734 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc00eae0 con 0x5592859260d0 2024-04-13T03:34:33.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40ce000640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f40b80a3bb0 0x7f40b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40ce000640 1 -- 172.21.15.16:0/2601436393 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f40c40d0fd0 con 0x7f40d0104220 2024-04-13T03:34:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40d6200640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f40b80a3bb0 0x7f40b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40db1f4740 1 -- 172.21.15.16:0/1599799734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 msgr2=0x559285928560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285928560 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009ca0 tx=0x7f40bc011810 comp rx=0 tx=0).stop 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40db1f4740 1 -- 172.21.15.16:0/1599799734 shutdown_connections 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285928560 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x559285925a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1599799734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 0x559285920990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.339+0000 7f40db1f4740 1 -- 172.21.15.16:0/1599799734 >> 172.21.15.16:0/1599799734 conn(0x55928591bae0 msgr2=0x55928591bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.340+0000 7f40db1f4740 1 -- 172.21.15.16:0/1599799734 shutdown_connections 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.340+0000 7f40db1f4740 1 -- 172.21.15.16:0/1599799734 wait complete. 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.340+0000 7f40d6200640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f40b80a3bb0 0x7f40b80a6070 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f40c8036040 tx=0x7f40c802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.340+0000 7f40db1f4740 1 -- start start 2024-04-13T03:34:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.340+0000 7f40db1f4740 1 Processor -- start 2024-04-13T03:34:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 0x55928592aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285931fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40db1f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559285929080 con 0x559285920f60 2024-04-13T03:34:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40db1f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5592858408f0 con 0x5592859260d0 2024-04-13T03:34:33.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40db1f4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559285929200 con 0x559285920570 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40db1f4740 10 client.0 _ll_register_callbacks cb 0x5592858402e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40d4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40d4e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38994/0 (socket says 172.21.15.16:38994) 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40d4e00640 1 -- 172.21.15.16:0/2458739746 learned_addr learned my addr 172.21.15.16:0/2458739746 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40d6200640 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285931fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.341+0000 7f40d5800640 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 0x55928592aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.342+0000 7f40d4e00640 1 -- 172.21.15.16:0/2458739746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 msgr2=0x55928592aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.342+0000 7f40d4e00640 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 0x55928592aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.342+0000 7f40d4e00640 1 -- 172.21.15.16:0/2458739746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 msgr2=0x559285931fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.342+0000 7f40d4e00640 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285931fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.342+0000 7f40d4e00640 1 -- 172.21.15.16:0/2458739746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592856636b0 con 0x559285920f60 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.342+0000 7f40d4e00640 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592af80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009710 tx=0x7f40bc019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b64ce52e0d6ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.342+0000 7f40b7400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.342+0000 7f40b7400640 1 -- 172.21.15.16:0/2458739746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc028020 con 0x559285920f60 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.343+0000 7f40db1f4740 1 -- 172.21.15.16:0/2458739746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40bc019b10 con 0x559285920f60 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.343+0000 7f40b7400640 1 -- 172.21.15.16:0/2458739746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f40bc005050 con 0x559285920f60 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.343+0000 7f40b7400640 1 -- 172.21.15.16:0/2458739746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40bc033050 con 0x559285920f60 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.343+0000 7f40db1f4740 10 client.15282 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.343+0000 7f40db1f4740 1 -- 172.21.15.16:0/2458739746 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559285935330 con 0x559285920f60 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.343+0000 7f40b7400640 1 -- 172.21.15.16:0/2458739746 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40bc019b10 con 0x559285920f60 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.344+0000 7f40b7400640 1 -- 172.21.15.16:0/2458739746 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f40bc0046c0 con 0x559285920f60 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.344+0000 7f40db1f4740 10 client.15282 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.344+0000 7f40db1f4740 10 client.15282 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-13T03:34:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.344+0000 7f40db1f4740 20 client.15282 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.344+0000 7f40db1f4740 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x5592859376c0 0x559285938b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.344+0000 7f40db1f4740 4 client.15282 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:34:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.344+0000 7f40db1f4740 1 -- 172.21.15.16:0/2458739746 --> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55928582cb40 con 0x5592859376c0 2024-04-13T03:34:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.344+0000 7f40d5800640 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x5592859376c0 0x559285938b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.345+0000 7f40d5800640 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x5592859376c0 0x559285938b30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=23e2111d0e1b717d server_cookie=f9b235495ecbd602 in_seq=0 out_seq=0 2024-04-13T03:34:33.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.345+0000 7f40b7400640 10 client.15282 ms_handle_connect on v2:172.21.15.188:6836/284865646 2024-04-13T03:34:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.345+0000 7f40b7400640 1 -- 172.21.15.16:0/2458739746 <== mds.3 v2:172.21.15.188:6836/284865646 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55928582cb40 con 0x5592859376c0 2024-04-13T03:34:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.345+0000 7f40b7400640 10 client.15282 handle_command_reply: tid=0 2024-04-13T03:34:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 client.15282 shutdown 2024-04-13T03:34:33.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 -- 172.21.15.16:0/2458739746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 msgr2=0x55928592af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592af80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009710 tx=0x7f40bc019660 comp rx=0 tx=0).stop 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 -- 172.21.15.16:0/2458739746 shutdown_connections 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x5592859376c0 0x559285938b30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285931fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592af80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 --2- 172.21.15.16:0/2458739746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920570 0x55928592aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40b7400640 0 client.15282 ms_handle_reset on v2:172.21.15.188:6836/284865646 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 -- 172.21.15.16:0/2458739746 >> 172.21.15.16:0/2458739746 conn(0x55928591bae0 msgr2=0x55928591ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 -- 172.21.15.16:0/2458739746 shutdown_connections 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 1 -- 172.21.15.16:0/2458739746 wait complete. 2024-04-13T03:34:33.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.346+0000 7f40db1f4740 20 client.15282 trim_cache size 0 max 16384 2024-04-13T03:34:33.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.367+0000 7f40db1f4740 1 Processor -- start 2024-04-13T03:34:33.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.367+0000 7f40db1f4740 1 -- start start 2024-04-13T03:34:33.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.367+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920570 0x559285938460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.367+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920f60 0x559285931f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.367+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285916ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.367+0000 7f40db1f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x559285936e70 con 0x559285920570 2024-04-13T03:34:33.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.367+0000 7f40db1f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559285937540 con 0x5592859260d0 2024-04-13T03:34:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.367+0000 7f40db1f4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559285937c10 con 0x559285920f60 2024-04-13T03:34:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d5800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920570 0x559285938460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d6200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285916ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d5800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920570 0x559285938460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38996/0 (socket says 172.21.15.16:38996) 2024-04-13T03:34:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d5800640 1 -- 172.21.15.16:0/1365765952 learned_addr learned my addr 172.21.15.16:0/1365765952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d4e00640 1 --2- 172.21.15.16:0/1365765952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920f60 0x559285931f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d5800640 1 -- 172.21.15.16:0/1365765952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920f60 msgr2=0x559285931f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d5800640 1 --2- 172.21.15.16:0/1365765952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920f60 0x559285931f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d5800640 1 -- 172.21.15.16:0/1365765952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 msgr2=0x559285916ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d5800640 1 --2- 172.21.15.16:0/1365765952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285916ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d5800640 1 -- 172.21.15.16:0/1365765952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559285921340 con 0x559285920570 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.368+0000 7f40d5800640 1 --2- 172.21.15.16:0/1365765952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920570 0x559285938460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f40c40049d0 tx=0x7f40c400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e05c4c7a15ea0db server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40c2000640 1 -- 172.21.15.16:0/1365765952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c409d5e0 con 0x559285920570 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40c2000640 1 -- 172.21.15.16:0/1365765952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c409cee0 con 0x559285920570 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40c2000640 1 -- 172.21.15.16:0/1365765952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4099550 con 0x559285920570 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40db1f4740 1 -- 172.21.15.16:0/1365765952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920570 msgr2=0x559285938460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1365765952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920570 0x559285938460 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f40c40049d0 tx=0x7f40c400cf90 comp rx=0 tx=0).stop 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40db1f4740 1 -- 172.21.15.16:0/1365765952 shutdown_connections 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1365765952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x559285916ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1365765952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285920f60 0x559285931f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1365765952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920570 0x559285938460 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40db1f4740 1 -- 172.21.15.16:0/1365765952 >> 172.21.15.16:0/1365765952 conn(0x55928591bae0 msgr2=0x55928591f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40db1f4740 1 -- 172.21.15.16:0/1365765952 shutdown_connections 2024-04-13T03:34:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.369+0000 7f40db1f4740 1 -- 172.21.15.16:0/1365765952 wait complete. 2024-04-13T03:34:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.370+0000 7f40db1f4740 1 -- start start 2024-04-13T03:34:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.370+0000 7f40db1f4740 1 Processor -- start 2024-04-13T03:34:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.370+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.370+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x55928592e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.370+0000 7f40db1f4740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285931630 0x559285931ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.370+0000 7f40db1f4740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x5592859389b0 con 0x559285920f60 2024-04-13T03:34:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.370+0000 7f40db1f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5592859382e0 con 0x5592859260d0 2024-04-13T03:34:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.370+0000 7f40db1f4740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x559285939080 con 0x559285931630 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.370+0000 7f40db1f4740 10 client.0 _ll_register_callbacks cb 0x7f40bc009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d4e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x55928592e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d4e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x55928592e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56144/0 (socket says 172.21.15.16:56144) 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d4e00640 1 -- 172.21.15.16:0/1916216284 learned_addr learned my addr 172.21.15.16:0/1916216284 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d5800640 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d6200640 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285931630 0x559285931ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d4e00640 1 -- 172.21.15.16:0/1916216284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285931630 msgr2=0x559285931ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d4e00640 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285931630 0x559285931ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d4e00640 1 -- 172.21.15.16:0/1916216284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 msgr2=0x55928592e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d4e00640 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d4e00640 1 -- 172.21.15.16:0/1916216284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559285932020 con 0x5592859260d0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40d4e00640 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x55928592e6f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009a30 tx=0x7f40bc01a910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9b11c7ccec8fecc server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.371+0000 7f40b7400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40b7400640 1 -- 172.21.15.16:0/1916216284 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc028040 con 0x5592859260d0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40db1f4740 1 -- 172.21.15.16:0/1916216284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40c409cee0 con 0x5592859260d0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40b7400640 1 -- 172.21.15.16:0/1916216284 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f40bc004250 con 0x5592859260d0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40b7400640 1 -- 172.21.15.16:0/1916216284 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40bc01e040 con 0x5592859260d0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40db1f4740 10 client.25394 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40db1f4740 1 -- 172.21.15.16:0/1916216284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5592859a6c90 con 0x5592859260d0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40b7400640 1 -- 172.21.15.16:0/1916216284 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c409cee0 con 0x5592859260d0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40b7400640 1 -- 172.21.15.16:0/1916216284 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f40bc003c00 con 0x5592859260d0 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40db1f4740 10 client.25394 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:33.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40db1f4740 10 client.25394 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40db1f4740 20 client.25394 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x5592859ab810 0x5592859aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40db1f4740 4 client.25394 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.372+0000 7f40db1f4740 1 -- 172.21.15.16:0/1916216284 --> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] -- command(tid 0: {"prefix": "status"}) -- 0x5592858943b0 con 0x5592859ab810 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.373+0000 7f40d5800640 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x5592859ab810 0x5592859aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.373+0000 7f40d5800640 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x5592859ab810 0x5592859aa520 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a22a78481bd65058 server_cookie=5c0539e7c625da0d in_seq=0 out_seq=0 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.373+0000 7f40b7400640 10 client.25394 ms_handle_connect on v2:172.21.15.188:6836/284865646 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40b7400640 1 -- 172.21.15.16:0/1916216284 <== mds.3 v2:172.21.15.188:6836/284865646 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5592858943b0 con 0x5592859ab810 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40b7400640 10 client.25394 handle_command_reply: tid=0 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 client.25394 shutdown 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 -- 172.21.15.16:0/1916216284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 msgr2=0x55928592e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x55928592e6f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009a30 tx=0x7f40bc01a910 comp rx=0 tx=0).stop 2024-04-13T03:34:33.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 -- 172.21.15.16:0/1916216284 shutdown_connections 2024-04-13T03:34:33.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.188:6836/284865646,v1:172.21.15.188:6837/284865646] conn(0x5592859ab810 0x5592859aa520 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x559285931630 0x559285931ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592859260d0 0x55928592e6f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40b7400640 0 client.25394 ms_handle_reset on v2:172.21.15.188:6836/284865646 2024-04-13T03:34:33.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 --2- 172.21.15.16:0/1916216284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x559285920f60 0x55928592e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 -- 172.21.15.16:0/1916216284 >> 172.21.15.16:0/1916216284 conn(0x55928591bae0 msgr2=0x55928591f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 -- 172.21.15.16:0/1916216284 shutdown_connections 2024-04-13T03:34:33.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 1 -- 172.21.15.16:0/1916216284 wait complete. 2024-04-13T03:34:33.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.374+0000 7f40db1f4740 20 client.25394 trim_cache size 0 max 16384 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 3, 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24505, 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 147.31522250399999, 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:34:33.381 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:34:33.382 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:34:33.382 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 199.48713689100001 2024-04-13T03:34:33.382 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:34:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 -- 172.21.15.16:0/2601436393 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f40b80a3bb0 msgr2=0x7f40b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f40b80a3bb0 0x7f40b80a6070 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f40c8036040 tx=0x7f40c802df90 comp rx=0 tx=0).stop 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 -- 172.21.15.16:0/2601436393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0104220 msgr2=0x7f40d019a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0104220 0x7f40d019a4c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f40c400ebc0 tx=0x7f40c400c6a0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 -- 172.21.15.16:0/2601436393 shutdown_connections 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f40b80a3bb0 0x7f40b80a6070 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d0105dd0 0x7f40d01a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40d0105410 0x7f40d019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 --2- 172.21.15.16:0/2601436393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40d0104220 0x7f40d019a4c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 -- 172.21.15.16:0/2601436393 >> 172.21.15.16:0/2601436393 conn(0x7f40d00ff7b0 msgr2=0x7f40d01086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 -- 172.21.15.16:0/2601436393 shutdown_connections 2024-04-13T03:34:33.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.375+0000 7f40d8200640 1 -- 172.21.15.16:0/2601436393 wait complete. 2024-04-13T03:34:33.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-13T03:34:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.606+0000 7f7d11e00640 1 -- 172.21.15.16:0/724465740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0c1052f0 msgr2=0x7f7d0c105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.606+0000 7f7d11e00640 1 --2- 172.21.15.16:0/724465740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0c1052f0 0x7f7d0c105770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009860 tx=0x7f7cf402bb70 comp rx=0 tx=0).stop 2024-04-13T03:34:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.607+0000 7f7d11e00640 1 -- 172.21.15.16:0/724465740 shutdown_connections 2024-04-13T03:34:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.607+0000 7f7d11e00640 1 --2- 172.21.15.16:0/724465740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d0c105cb0 0x7f7d0c10c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.607+0000 7f7d11e00640 1 --2- 172.21.15.16:0/724465740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0c1052f0 0x7f7d0c105770 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.607+0000 7f7d11e00640 1 --2- 172.21.15.16:0/724465740 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d0c104100 0x7f7d0c104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.607+0000 7f7d11e00640 1 -- 172.21.15.16:0/724465740 >> 172.21.15.16:0/724465740 conn(0x7f7d0c0ff670 msgr2=0x7f7d0c101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.607+0000 7f7d11e00640 1 -- 172.21.15.16:0/724465740 shutdown_connections 2024-04-13T03:34:33.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.607+0000 7f7d11e00640 1 -- 172.21.15.16:0/724465740 wait complete. 2024-04-13T03:34:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d11e00640 1 Processor -- start 2024-04-13T03:34:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d11e00640 1 -- start start 2024-04-13T03:34:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d11e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d0c104100 0x7f7d0c19a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d11e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d0c1052f0 0x7f7d0c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d11e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0c105cb0 0x7f7d0c1a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d11e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c10fd90 con 0x7f7d0c104100 2024-04-13T03:34:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d11e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c10fc10 con 0x7f7d0c1052f0 2024-04-13T03:34:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d11e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c10ff10 con 0x7f7d0c105cb0 2024-04-13T03:34:33.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d03e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d0c1052f0 0x7f7d0c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d03e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d0c1052f0 0x7f7d0c19a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56152/0 (socket says 172.21.15.16:56152) 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d03e00640 1 -- 172.21.15.16:0/148378207 learned_addr learned my addr 172.21.15.16:0/148378207 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.608+0000 7f7d10a00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d0c104100 0x7f7d0c19a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d11400640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0c105cb0 0x7f7d0c1a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d03e00640 1 -- 172.21.15.16:0/148378207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0c105cb0 msgr2=0x7f7d0c1a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d03e00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0c105cb0 0x7f7d0c1a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d03e00640 1 -- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d0c104100 msgr2=0x7f7d0c19a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d03e00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d0c104100 0x7f7d0c19a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d03e00640 1 -- 172.21.15.16:0/148378207 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0c1a2010 con 0x7f7d0c1052f0 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d10a00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d0c104100 0x7f7d0c19a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d11400640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0c105cb0 0x7f7d0c1a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d03e00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d0c1052f0 0x7f7d0c19a930 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40349e0 tx=0x7f7cf4032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d0a000640 1 -- 172.21.15.16:0/148378207 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4041030 con 0x7f7d0c1052f0 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d0a000640 1 -- 172.21.15.16:0/148378207 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cf402c510 con 0x7f7d0c1052f0 2024-04-13T03:34:33.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d11e00640 1 -- 172.21.15.16:0/148378207 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d0c1a22a0 con 0x7f7d0c1052f0 2024-04-13T03:34:33.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.609+0000 7f7d0a000640 1 -- 172.21.15.16:0/148378207 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4039040 con 0x7f7d0c1052f0 2024-04-13T03:34:33.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.610+0000 7f7d11e00640 1 -- 172.21.15.16:0/148378207 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d0c1a2790 con 0x7f7d0c1052f0 2024-04-13T03:34:33.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.613+0000 7f7d0a000640 1 -- 172.21.15.16:0/148378207 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7cf4044070 con 0x7f7d0c1052f0 2024-04-13T03:34:33.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.613+0000 7f7d16232740 1 Processor -- start 2024-04-13T03:34:33.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.613+0000 7f7d16232740 1 -- start start 2024-04-13T03:34:33.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.613+0000 7f7d16232740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4e2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.613+0000 7f7d16232740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 0x558ead4e7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.613+0000 7f7d16232740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e8310 0x558ead4ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.613+0000 7f7d16232740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558ead448990 con 0x558ead4e8310 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.613+0000 7f7d16232740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558ead3ef530 con 0x558ead4e27b0 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.613+0000 7f7d16232740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558ead444950 con 0x558ead4e31a0 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d11400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4e2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d11400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4e2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56168/0 (socket says 172.21.15.16:56168) 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d11400640 1 -- 172.21.15.16:0/509573800 learned_addr learned my addr 172.21.15.16:0/509573800 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d03e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e8310 0x558ead4ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d10a00640 1 --2- 172.21.15.16:0/509573800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 0x558ead4e7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d11400640 1 -- 172.21.15.16:0/509573800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 msgr2=0x558ead4e7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d11400640 1 --2- 172.21.15.16:0/509573800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 0x558ead4e7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d11400640 1 -- 172.21.15.16:0/509573800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e8310 msgr2=0x558ead4ea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d11400640 1 --2- 172.21.15.16:0/509573800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e8310 0x558ead4ea7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d11400640 1 -- 172.21.15.16:0/509573800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ead15b2d0 con 0x558ead4e27b0 2024-04-13T03:34:33.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.614+0000 7f7d0a000640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7cf00a3bb0 0x7f7cf00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.615+0000 7f7d11400640 1 --2- 172.21.15.16:0/509573800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4e2bd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009c70 tx=0x7f7d040117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4a2da002ca45e30 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.615+0000 7f7d0a000640 1 -- 172.21.15.16:0/148378207 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7cf40f1e60 con 0x7f7d0c1052f0 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.615+0000 7f7d10a00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7cf00a3bb0 0x7f7cf00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.615+0000 7f7d01c00640 1 -- 172.21.15.16:0/509573800 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04012450 con 0x558ead4e27b0 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.615+0000 7f7d01c00640 1 -- 172.21.15.16:0/509573800 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d04012ad0 con 0x558ead4e27b0 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.615+0000 7f7d01c00640 1 -- 172.21.15.16:0/509573800 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0400fc60 con 0x558ead4e27b0 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 -- 172.21.15.16:0/509573800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 msgr2=0x558ead4e2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d10a00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7cf00a3bb0 0x7f7cf00a6070 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x558ead4e35a0 tx=0x7f7cfc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 --2- 172.21.15.16:0/509573800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4e2bd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009c70 tx=0x7f7d040117b0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 -- 172.21.15.16:0/509573800 shutdown_connections 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 --2- 172.21.15.16:0/509573800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e8310 0x558ead4ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 --2- 172.21.15.16:0/509573800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 0x558ead4e7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 --2- 172.21.15.16:0/509573800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4e2bd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 -- 172.21.15.16:0/509573800 >> 172.21.15.16:0/509573800 conn(0x558ead4ddd20 msgr2=0x558ead4de120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 -- 172.21.15.16:0/509573800 shutdown_connections 2024-04-13T03:34:33.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 -- 172.21.15.16:0/509573800 wait complete. 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.616+0000 7f7d16232740 1 -- start start 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.617+0000 7f7d16232740 1 Processor -- start 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.617+0000 7f7d16232740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4ecd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.617+0000 7f7d16232740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e31a0 0x558ead4ed270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.617+0000 7f7d16232740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 0x558ead4f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.617+0000 7f7d16232740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558ead4eb2c0 con 0x558ead4e31a0 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.617+0000 7f7d16232740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558ead4028f0 con 0x558ead4e27b0 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.617+0000 7f7d16232740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558ead4eb440 con 0x558ead4e8310 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.617+0000 7f7d16232740 10 client.0 _ll_register_callbacks cb 0x558ead4022e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d10a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4ecd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d11400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 0x558ead4f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d10a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4ecd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56170/0 (socket says 172.21.15.16:56170) 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d11400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 0x558ead4f42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54104/0 (socket says 172.21.15.16:54104) 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d10a00640 1 -- 172.21.15.16:0/3763836424 learned_addr learned my addr 172.21.15.16:0/3763836424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d03e00640 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e31a0 0x558ead4ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d10a00640 1 -- 172.21.15.16:0/3763836424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 msgr2=0x558ead4f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d10a00640 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 0x558ead4f42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d10a00640 1 -- 172.21.15.16:0/3763836424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e31a0 msgr2=0x558ead4ed270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d10a00640 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e31a0 0x558ead4ed270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d10a00640 1 -- 172.21.15.16:0/3763836424 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ead2256b0 con 0x558ead4e27b0 2024-04-13T03:34:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d03e00640 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e31a0 0x558ead4ed270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7d10a00640 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4ecd30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc01f9e0 tx=0x7f7cfc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa50724565bd547b server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.618+0000 7f7cfa000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-13T03:34:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.622+0000 7f7cfa000640 1 -- 172.21.15.16:0/3763836424 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc004b20 con 0x558ead4e27b0 2024-04-13T03:34:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.622+0000 7f7cfa000640 1 -- 172.21.15.16:0/3763836424 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7cfc017030 con 0x558ead4e27b0 2024-04-13T03:34:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7d16232740 1 -- 172.21.15.16:0/3763836424 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d04012ad0 con 0x558ead4e27b0 2024-04-13T03:34:33.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7cfa000640 1 -- 172.21.15.16:0/3763836424 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7cfc004cc0 con 0x558ead4e27b0 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7d16232740 10 client.25406 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7d16232740 1 -- 172.21.15.16:0/3763836424 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558ead4f7690 con 0x558ead4e27b0 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7cfa000640 1 -- 172.21.15.16:0/3763836424 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d04012ad0 con 0x558ead4e27b0 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7cfa000640 1 -- 172.21.15.16:0/3763836424 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f7cfc0055d0 con 0x558ead4e27b0 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7d16232740 10 client.25406 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7d16232740 10 client.25406 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7d16232740 20 client.25406 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7d16232740 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x558ead4f9cf0 0x558ead4fb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7d16232740 4 client.25406 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:34:33.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.623+0000 7f7d16232740 1 -- 172.21.15.16:0/3763836424 --> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ead3eeb40 con 0x558ead4f9cf0 2024-04-13T03:34:33.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.624+0000 7f7d03e00640 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x558ead4f9cf0 0x558ead4fb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.625+0000 7f7d03e00640 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x558ead4f9cf0 0x558ead4fb160 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e490a67b1704cd24 server_cookie=e017df6cfe4602 in_seq=0 out_seq=0 2024-04-13T03:34:33.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.625+0000 7f7cfa000640 10 client.25406 ms_handle_connect on v2:172.21.15.188:6838/57242391 2024-04-13T03:34:33.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.625+0000 7f7cfa000640 1 -- 172.21.15.16:0/3763836424 <== mds.2 v2:172.21.15.188:6838/57242391 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ead3eeb40 con 0x558ead4f9cf0 2024-04-13T03:34:33.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.625+0000 7f7cfa000640 10 client.25406 handle_command_reply: tid=0 2024-04-13T03:34:33.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 client.25406 shutdown 2024-04-13T03:34:33.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 -- 172.21.15.16:0/3763836424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 msgr2=0x558ead4ecd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4ecd30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc01f9e0 tx=0x7f7cfc004a40 comp rx=0 tx=0).stop 2024-04-13T03:34:33.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 -- 172.21.15.16:0/3763836424 shutdown_connections 2024-04-13T03:34:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 0x558ead4f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e31a0 0x558ead4ed270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x558ead4f9cf0 0x558ead4fb160 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 --2- 172.21.15.16:0/3763836424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e27b0 0x558ead4ecd30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 -- 172.21.15.16:0/3763836424 >> 172.21.15.16:0/3763836424 conn(0x558ead4ddd20 msgr2=0x558ead4dcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7cfa000640 0 client.25406 ms_handle_reset on v2:172.21.15.188:6838/57242391 2024-04-13T03:34:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 -- 172.21.15.16:0/3763836424 shutdown_connections 2024-04-13T03:34:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 1 -- 172.21.15.16:0/3763836424 wait complete. 2024-04-13T03:34:33.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.626+0000 7f7d16232740 20 client.25406 trim_cache size 0 max 16384 2024-04-13T03:34:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.646+0000 7f7d16232740 1 Processor -- start 2024-04-13T03:34:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.646+0000 7f7d16232740 1 -- start start 2024-04-13T03:34:33.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d16232740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4ea700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d16232740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e31a0 0x558ead4f4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d16232740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 0x558ead4e3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d16232740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558ead4f94a0 con 0x558ead4e27b0 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d16232740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558ead4f9b70 con 0x558ead4e31a0 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d16232740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558ead4fa240 con 0x558ead4e8310 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d10a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d10a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4ea700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39038/0 (socket says 172.21.15.16:39038) 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d03e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e31a0 0x558ead4f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d10a00640 1 -- 172.21.15.16:0/1195470727 learned_addr learned my addr 172.21.15.16:0/1195470727 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.647+0000 7f7d11400640 1 --2- 172.21.15.16:0/1195470727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 0x558ead4e3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d10a00640 1 -- 172.21.15.16:0/1195470727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 msgr2=0x558ead4e3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d10a00640 1 --2- 172.21.15.16:0/1195470727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 0x558ead4e3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d10a00640 1 -- 172.21.15.16:0/1195470727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e31a0 msgr2=0x558ead4f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d10a00640 1 --2- 172.21.15.16:0/1195470727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e31a0 0x558ead4f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d10a00640 1 -- 172.21.15.16:0/1195470727 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ead4edeb0 con 0x558ead4e27b0 2024-04-13T03:34:33.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d03e00640 1 --2- 172.21.15.16:0/1195470727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e31a0 0x558ead4f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d10a00640 1 --2- 172.21.15.16:0/1195470727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4ea700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009600 tx=0x7f7cfc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7034486c302dee8e server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d01200640 1 -- 172.21.15.16:0/1195470727 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc039070 con 0x558ead4e27b0 2024-04-13T03:34:33.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d01200640 1 -- 172.21.15.16:0/1195470727 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cfc01c6b0 con 0x558ead4e27b0 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.648+0000 7f7d01200640 1 -- 172.21.15.16:0/1195470727 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc0050c0 con 0x558ead4e27b0 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 -- 172.21.15.16:0/1195470727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 msgr2=0x558ead4ea700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 --2- 172.21.15.16:0/1195470727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4ea700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009600 tx=0x7f7cfc003970 comp rx=0 tx=0).stop 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 -- 172.21.15.16:0/1195470727 shutdown_connections 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 --2- 172.21.15.16:0/1195470727 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e8310 0x558ead4e3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 --2- 172.21.15.16:0/1195470727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e31a0 0x558ead4f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 --2- 172.21.15.16:0/1195470727 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4ea700 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 -- 172.21.15.16:0/1195470727 >> 172.21.15.16:0/1195470727 conn(0x558ead4ddd20 msgr2=0x558ead4eb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 -- 172.21.15.16:0/1195470727 shutdown_connections 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 -- 172.21.15.16:0/1195470727 wait complete. 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.649+0000 7f7d16232740 1 -- start start 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d16232740 1 Processor -- start 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d16232740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4f2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d16232740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 0x558ead55bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d16232740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e8310 0x558ead55e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d16232740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558ead4fafe0 con 0x558ead4e27b0 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d16232740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558ead4fa910 con 0x558ead4e8310 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d16232740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x558ead4fb6b0 con 0x558ead4e31a0 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d16232740 10 client.0 _ll_register_callbacks cb 0x7f7cfc01f8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d11400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e8310 0x558ead55e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d11400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e8310 0x558ead55e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56200/0 (socket says 172.21.15.16:56200) 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d11400640 1 -- 172.21.15.16:0/2874095413 learned_addr learned my addr 172.21.15.16:0/2874095413 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d10a00640 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.650+0000 7f7d03e00640 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 0x558ead55bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7d11400640 1 -- 172.21.15.16:0/2874095413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 msgr2=0x558ead55bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7d11400640 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 0x558ead55bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7d11400640 1 -- 172.21.15.16:0/2874095413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 msgr2=0x558ead4f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7d11400640 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4f2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7d11400640 1 -- 172.21.15.16:0/2874095413 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ead55e790 con 0x558ead4e8310 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7d10a00640 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7d11400640 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e8310 0x558ead55e1d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009bc0 tx=0x7f7d040045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd2f116645854a60 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7cfa000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7cfa000640 1 -- 172.21.15.16:0/2874095413 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04005450 con 0x558ead4e8310 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7cfa000640 1 -- 172.21.15.16:0/2874095413 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7d040055f0 con 0x558ead4e8310 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.651+0000 7f7d16232740 1 -- 172.21.15.16:0/2874095413 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cfc01c6b0 con 0x558ead4e8310 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7cfa000640 1 -- 172.21.15.16:0/2874095413 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7d04037050 con 0x558ead4e8310 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7d16232740 10 client.25412 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7d16232740 1 -- 172.21.15.16:0/2874095413 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558ead55fa70 con 0x558ead4e8310 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7cfa000640 1 -- 172.21.15.16:0/2874095413 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cfc01c6b0 con 0x558ead4e8310 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7cfa000640 1 -- 172.21.15.16:0/2874095413 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f7d04003920 con 0x558ead4e8310 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7d16232740 10 client.25412 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:33.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7d16232740 10 client.25412 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-13T03:34:33.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7d16232740 20 client.25412 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:33.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7d16232740 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x558ead564300 0x558ead560740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7d16232740 4 client.25412 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:34:33.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.652+0000 7f7d16232740 1 -- 172.21.15.16:0/2874095413 --> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] -- command(tid 0: {"prefix": "status"}) -- 0x558ead4563b0 con 0x558ead564300 2024-04-13T03:34:33.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.653+0000 7f7d10a00640 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x558ead564300 0x558ead560740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.653+0000 7f7d10a00640 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x558ead564300 0x558ead560740 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d41a0f5ea57bef4 server_cookie=2f6c39ea96ec713a in_seq=0 out_seq=0 2024-04-13T03:34:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.653+0000 7f7cfa000640 10 client.25412 ms_handle_connect on v2:172.21.15.188:6838/57242391 2024-04-13T03:34:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.653+0000 7f7cfa000640 1 -- 172.21.15.16:0/2874095413 <== mds.2 v2:172.21.15.188:6838/57242391 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558ead4563b0 con 0x558ead564300 2024-04-13T03:34:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.653+0000 7f7cfa000640 10 client.25412 handle_command_reply: tid=0 2024-04-13T03:34:33.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.653+0000 7f7d16232740 1 client.25412 shutdown 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.653+0000 7f7d16232740 1 -- 172.21.15.16:0/2874095413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e8310 msgr2=0x558ead55e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e8310 0x558ead55e1d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009bc0 tx=0x7f7d040045d0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 1 -- 172.21.15.16:0/2874095413 shutdown_connections 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.188:6838/57242391,v1:172.21.15.188:6839/57242391] conn(0x558ead564300 0x558ead560740 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558ead4e8310 0x558ead55e1d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558ead4e31a0 0x558ead55bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7cfa000640 0 client.25412 ms_handle_reset on v2:172.21.15.188:6838/57242391 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 1 --2- 172.21.15.16:0/2874095413 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558ead4e27b0 0x558ead4f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 1 -- 172.21.15.16:0/2874095413 >> 172.21.15.16:0/2874095413 conn(0x558ead4ddd20 msgr2=0x558ead4e3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 1 -- 172.21.15.16:0/2874095413 shutdown_connections 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 1 -- 172.21.15.16:0/2874095413 wait complete. 2024-04-13T03:34:33.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d16232740 20 client.25412 trim_cache size 0 max 16384 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 2, 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24523, 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 148.61720071100001, 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 193.96122297599999 2024-04-13T03:34:33.659 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:34:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d11e00640 1 -- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7cf00a3bb0 msgr2=0x7f7cf00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d11e00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7cf00a3bb0 0x7f7cf00a6070 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x558ead4e35a0 tx=0x7f7cfc009500 comp rx=0 tx=0).stop 2024-04-13T03:34:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d11e00640 1 -- 172.21.15.16:0/148378207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d0c1052f0 msgr2=0x7f7d0c19a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.654+0000 7f7d11e00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d0c1052f0 0x7f7d0c19a930 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40349e0 tx=0x7f7cf4032450 comp rx=0 tx=0).stop 2024-04-13T03:34:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.655+0000 7f7d11e00640 1 -- 172.21.15.16:0/148378207 shutdown_connections 2024-04-13T03:34:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.655+0000 7f7d11e00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f7cf00a3bb0 0x7f7cf00a6070 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.655+0000 7f7d11e00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d0c105cb0 0x7f7d0c1a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.655+0000 7f7d11e00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d0c1052f0 0x7f7d0c19a930 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.655+0000 7f7d11e00640 1 --2- 172.21.15.16:0/148378207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d0c104100 0x7f7d0c19a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.655+0000 7f7d11e00640 1 -- 172.21.15.16:0/148378207 >> 172.21.15.16:0/148378207 conn(0x7f7d0c0ff670 msgr2=0x7f7d0c1085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.655+0000 7f7d11e00640 1 -- 172.21.15.16:0/148378207 shutdown_connections 2024-04-13T03:34:33.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.655+0000 7f7d11e00640 1 -- 172.21.15.16:0/148378207 wait complete. 2024-04-13T03:34:33.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-13T03:34:33.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:33 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2607638959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:34:33.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:33 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2607638959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:34:33.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.901+0000 7f08bae00640 1 -- 172.21.15.16:0/1115032239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 msgr2=0x7f08b4104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.901+0000 7f08bae00640 1 --2- 172.21.15.16:0/1115032239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 0x7f08b4104520 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f08a8009900 tx=0x7f08a802bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:33.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.902+0000 7f08bae00640 1 -- 172.21.15.16:0/1115032239 shutdown_connections 2024-04-13T03:34:33.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.902+0000 7f08bae00640 1 --2- 172.21.15.16:0/1115032239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08b4105cb0 0x7f08b410c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.902+0000 7f08bae00640 1 --2- 172.21.15.16:0/1115032239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b41052f0 0x7f08b4105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.902+0000 7f08bae00640 1 --2- 172.21.15.16:0/1115032239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 0x7f08b4104520 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.902+0000 7f08bae00640 1 -- 172.21.15.16:0/1115032239 >> 172.21.15.16:0/1115032239 conn(0x7f08b40ff670 msgr2=0x7f08b4101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.902+0000 7f08bae00640 1 -- 172.21.15.16:0/1115032239 shutdown_connections 2024-04-13T03:34:33.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.902+0000 7f08bae00640 1 -- 172.21.15.16:0/1115032239 wait complete. 2024-04-13T03:34:33.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.903+0000 7f08bae00640 1 Processor -- start 2024-04-13T03:34:33.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.903+0000 7f08bae00640 1 -- start start 2024-04-13T03:34:33.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.903+0000 7f08bae00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 0x7f08b419a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.903+0000 7f08bae00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b41052f0 0x7f08b419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.903+0000 7f08bae00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08b4105cb0 0x7f08b41a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.903+0000 7f08bae00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08b410fde0 con 0x7f08b41052f0 2024-04-13T03:34:33.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.903+0000 7f08bae00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f08b410fc60 con 0x7f08b4104100 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.903+0000 7f08bae00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f08b410ff60 con 0x7f08b4105cb0 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 0x7f08b419a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 0x7f08b419a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56224/0 (socket says 172.21.15.16:56224) 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9a00640 1 -- 172.21.15.16:0/1495150387 learned_addr learned my addr 172.21.15.16:0/1495150387 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9000640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b41052f0 0x7f08b419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08ba400640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08b4105cb0 0x7f08b41a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9a00640 1 -- 172.21.15.16:0/1495150387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08b4105cb0 msgr2=0x7f08b41a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9a00640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08b4105cb0 0x7f08b41a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9a00640 1 -- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b41052f0 msgr2=0x7f08b419a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9a00640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b41052f0 0x7f08b419a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9a00640 1 -- 172.21.15.16:0/1495150387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b41a2010 con 0x7f08b4104100 2024-04-13T03:34:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9000640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b41052f0 0x7f08b419a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.904+0000 7f08b9a00640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 0x7f08b419a3f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f08a8000c00 tx=0x7f08a8005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.905+0000 7f08b2000640 1 -- 172.21.15.16:0/1495150387 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8043020 con 0x7f08b4104100 2024-04-13T03:34:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.905+0000 7f08b2000640 1 -- 172.21.15.16:0/1495150387 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a80054e0 con 0x7f08b4104100 2024-04-13T03:34:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.905+0000 7f08bae00640 1 -- 172.21.15.16:0/1495150387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b41a22a0 con 0x7f08b4104100 2024-04-13T03:34:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.905+0000 7f08b2000640 1 -- 172.21.15.16:0/1495150387 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8003e60 con 0x7f08b4104100 2024-04-13T03:34:33.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.905+0000 7f08bae00640 1 -- 172.21.15.16:0/1495150387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b41a2840 con 0x7f08b4104100 2024-04-13T03:34:33.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.908+0000 7f08bf3e2740 1 Processor -- start 2024-04-13T03:34:33.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.908+0000 7f08b2000640 1 -- 172.21.15.16:0/1495150387 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f08a8046020 con 0x7f08b4104100 2024-04-13T03:34:33.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.908+0000 7f08bf3e2740 1 -- start start 2024-04-13T03:34:33.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.908+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2157b0 0x56487b215bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.908+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b21aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.908+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b21d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56487b17b990 con 0x56487b2157b0 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56487b122530 con 0x56487b21b310 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56487b177950 con 0x56487b2161a0 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08ba400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2157b0 0x56487b215bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b21aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08b9000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b21d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08ba400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2157b0 0x56487b215bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39064/0 (socket says 172.21.15.16:39064) 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08ba400640 1 -- 172.21.15.16:0/2142274917 learned_addr learned my addr 172.21.15.16:0/2142274917 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b21aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54166/0 (socket says 172.21.15.16:54166) 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08b9000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b21d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56238/0 (socket says 172.21.15.16:56238) 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08b9000640 1 -- 172.21.15.16:0/2142274917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 msgr2=0x56487b21aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08b9000640 1 --2- 172.21.15.16:0/2142274917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b21aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08b9000640 1 -- 172.21.15.16:0/2142274917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2157b0 msgr2=0x56487b215bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08b9000640 1 --2- 172.21.15.16:0/2142274917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2157b0 0x56487b215bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08b9000640 1 -- 172.21.15.16:0/2142274917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56487ae8e2d0 con 0x56487b21b310 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.909+0000 7f08ba400640 1 --2- 172.21.15.16:0/2142274917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2157b0 0x56487b215bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.910+0000 7f08b2000640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f089c0a3c30 0x7f089c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.910+0000 7f08b9000640 1 --2- 172.21.15.16:0/2142274917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b21d7a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f08a4005fd0 tx=0x7f08a4010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4dc900c857a7dd server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.910+0000 7f08b2000640 1 -- 172.21.15.16:0/1495150387 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f08a80f2a30 con 0x7f08b4104100 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.910+0000 7f08ba400640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f089c0a3c30 0x7f089c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.910+0000 7f08a2a00640 1 -- 172.21.15.16:0/2142274917 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a4016070 con 0x56487b21b310 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.910+0000 7f08a2a00640 1 -- 172.21.15.16:0/2142274917 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a401bb40 con 0x56487b21b310 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.910+0000 7f08a2a00640 1 -- 172.21.15.16:0/2142274917 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a4012d10 con 0x56487b21b310 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2142274917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 msgr2=0x56487b21d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2142274917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b21d7a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f08a4005fd0 tx=0x7f08a4010800 comp rx=0 tx=0).stop 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08ba400640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f089c0a3c30 0x7f089c0a60f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f08ac005fd0 tx=0x7f08ac00b560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2142274917 shutdown_connections 2024-04-13T03:34:33.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2142274917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b21d7a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2142274917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b21aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2142274917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2157b0 0x56487b215bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2142274917 >> 172.21.15.16:0/2142274917 conn(0x56487b210d20 msgr2=0x56487b211120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2142274917 shutdown_connections 2024-04-13T03:34:33.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2142274917 wait complete. 2024-04-13T03:34:33.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.911+0000 7f08bf3e2740 1 -- start start 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.912+0000 7f08bf3e2740 1 Processor -- start 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.912+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b21fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.912+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b2201f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.912+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 0x56487b227230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.912+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56487b21e2c0 con 0x56487b21b310 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.912+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56487b1358f0 con 0x56487b2157b0 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.912+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56487b21e440 con 0x56487b2161a0 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.912+0000 7f08bf3e2740 10 client.0 _ll_register_callbacks cb 0x56487b1352e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.912+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b21fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b21fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56242/0 (socket says 172.21.15.16:56242) 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08b9a00640 1 -- 172.21.15.16:0/3341577405 learned_addr learned my addr 172.21.15.16:0/3341577405 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08ba400640 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 0x56487b227230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08b9000640 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b2201f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08b9a00640 1 -- 172.21.15.16:0/3341577405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 msgr2=0x56487b2201f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08b9a00640 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b2201f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08b9a00640 1 -- 172.21.15.16:0/3341577405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 msgr2=0x56487b227230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08b9a00640 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 0x56487b227230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08b9a00640 1 -- 172.21.15.16:0/3341577405 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56487af586b0 con 0x56487b2157b0 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08b9a00640 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b21fcb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f08a8006010 tx=0x7f08a80f6100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf9fdb1e315b964 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f0897400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f0897400640 1 -- 172.21.15.16:0/3341577405 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a80be480 con 0x56487b2157b0 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f0897400640 1 -- 172.21.15.16:0/3341577405 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f08a80035f0 con 0x56487b2157b0 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.913+0000 7f08bf3e2740 1 -- 172.21.15.16:0/3341577405 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08a401bb40 con 0x56487b2157b0 2024-04-13T03:34:33.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f0897400640 1 -- 172.21.15.16:0/3341577405 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f08a80be2a0 con 0x56487b2157b0 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f08bf3e2740 10 client.25424 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f08bf3e2740 1 -- 172.21.15.16:0/3341577405 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56487b22a5a0 con 0x56487b2157b0 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f0897400640 1 -- 172.21.15.16:0/3341577405 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a401bb40 con 0x56487b2157b0 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f0897400640 1 -- 172.21.15.16:0/3341577405 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f08a80beba0 con 0x56487b2157b0 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f08bf3e2740 10 client.25424 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f08bf3e2740 10 client.25424 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f08bf3e2740 20 client.25424 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x56487b22cc30 0x56487b22e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f08bf3e2740 4 client.25424 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f08bf3e2740 1 -- 172.21.15.16:0/3341577405 --> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56487b121b40 con 0x56487b22cc30 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.914+0000 7f08b9000640 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x56487b22cc30 0x56487b22e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.915+0000 7f08b9000640 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x56487b22cc30 0x56487b22e0a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d67c2db28b4af998 server_cookie=1afa0fc34259f656 in_seq=0 out_seq=0 2024-04-13T03:34:33.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.915+0000 7f0897400640 10 client.25424 ms_handle_connect on v2:172.21.15.119:6836/477256695 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f0897400640 1 -- 172.21.15.16:0/3341577405 <== mds.1 v2:172.21.15.119:6836/477256695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56487b121b40 con 0x56487b22cc30 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f0897400640 10 client.25424 handle_command_reply: tid=0 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 client.25424 shutdown 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 -- 172.21.15.16:0/3341577405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 msgr2=0x56487b21fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b21fcb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f08a8006010 tx=0x7f08a80f6100 comp rx=0 tx=0).stop 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 -- 172.21.15.16:0/3341577405 shutdown_connections 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 0x56487b227230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b2201f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x56487b22cc30 0x56487b22e0a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/3341577405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b21fcb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 -- 172.21.15.16:0/3341577405 >> 172.21.15.16:0/3341577405 conn(0x56487b210d20 msgr2=0x56487b20ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f0897400640 0 client.25424 ms_handle_reset on v2:172.21.15.119:6836/477256695 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.916+0000 7f08bf3e2740 1 -- 172.21.15.16:0/3341577405 shutdown_connections 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.917+0000 7f08bf3e2740 1 -- 172.21.15.16:0/3341577405 wait complete. 2024-04-13T03:34:33.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.917+0000 7f08bf3e2740 20 client.25424 trim_cache size 0 max 16384 2024-04-13T03:34:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.936+0000 7f08bf3e2740 1 Processor -- start 2024-04-13T03:34:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08bf3e2740 1 -- start start 2024-04-13T03:34:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b22b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b22be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 0x56487b2271f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56487b22c3e0 con 0x56487b21b310 2024-04-13T03:34:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56487b22cab0 con 0x56487b2157b0 2024-04-13T03:34:33.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56487b22d180 con 0x56487b2161a0 2024-04-13T03:34:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08ba400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 0x56487b2271f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b22b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b22b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56252/0 (socket says 172.21.15.16:56252) 2024-04-13T03:34:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.937+0000 7f08b9a00640 1 -- 172.21.15.16:0/929880815 learned_addr learned my addr 172.21.15.16:0/929880815 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08b9000640 1 --2- 172.21.15.16:0/929880815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b22be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08b9a00640 1 -- 172.21.15.16:0/929880815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 msgr2=0x56487b22be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08b9a00640 1 --2- 172.21.15.16:0/929880815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b22be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08b9a00640 1 -- 172.21.15.16:0/929880815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 msgr2=0x56487b2271f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08b9a00640 1 --2- 172.21.15.16:0/929880815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 0x56487b2271f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08b9a00640 1 -- 172.21.15.16:0/929880815 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56487b216580 con 0x56487b2157b0 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08b9a00640 1 --2- 172.21.15.16:0/929880815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b22b8f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f08a80bada0 tx=0x7f08a80045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38268821f2cd7cb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08a2000640 1 -- 172.21.15.16:0/929880815 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a80be480 con 0x56487b2157b0 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08a2000640 1 -- 172.21.15.16:0/929880815 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a80fa070 con 0x56487b2157b0 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.938+0000 7f08a2000640 1 -- 172.21.15.16:0/929880815 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a80035f0 con 0x56487b2157b0 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 -- 172.21.15.16:0/929880815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 msgr2=0x56487b22b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/929880815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b22b8f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f08a80bada0 tx=0x7f08a80045e0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 -- 172.21.15.16:0/929880815 shutdown_connections 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/929880815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b21b310 0x56487b2271f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/929880815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2161a0 0x56487b22be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/929880815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b2157b0 0x56487b22b8f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 -- 172.21.15.16:0/929880815 >> 172.21.15.16:0/929880815 conn(0x56487b210d20 msgr2=0x56487b211100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 -- 172.21.15.16:0/929880815 shutdown_connections 2024-04-13T03:34:33.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 -- 172.21.15.16:0/929880815 wait complete. 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.939+0000 7f08bf3e2740 1 -- start start 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08bf3e2740 1 Processor -- start 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2157b0 0x56487b2251e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2161a0 0x56487b28ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08bf3e2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b291110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56487b22df20 con 0x56487b2161a0 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56487b22d850 con 0x56487b21b310 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08bf3e2740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x56487b22e5f0 con 0x56487b2157b0 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08bf3e2740 10 client.0 _ll_register_callbacks cb 0x7f08a80df0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08b9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2161a0 0x56487b28ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2157b0 0x56487b2251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08b9000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2161a0 0x56487b28ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39096/0 (socket says 172.21.15.16:39096) 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08b9000640 1 -- 172.21.15.16:0/2668421480 learned_addr learned my addr 172.21.15.16:0/2668421480 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.940+0000 7f08ba400640 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b291110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f08b9000640 1 -- 172.21.15.16:0/2668421480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2157b0 msgr2=0x56487b2251e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f08b9000640 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2157b0 0x56487b2251e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f08b9000640 1 -- 172.21.15.16:0/2668421480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 msgr2=0x56487b291110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f08b9000640 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b291110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f08b9000640 1 -- 172.21.15.16:0/2668421480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56487b291650 con 0x56487b2161a0 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f08b9a00640 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2157b0 0x56487b2251e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f08b9000640 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2161a0 0x56487b28ebc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x56487b211c90 tx=0x7f08a4009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a335a0c41ec77942 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f0897400640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f0897400640 1 -- 172.21.15.16:0/2668421480 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a4016040 con 0x56487b2161a0 2024-04-13T03:34:33.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f0897400640 1 -- 172.21.15.16:0/2668421480 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f08a401b460 con 0x56487b2161a0 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.941+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2668421480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08a80fa070 con 0x56487b2161a0 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f0897400640 1 -- 172.21.15.16:0/2668421480 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f08a4034050 con 0x56487b2161a0 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f08bf3e2740 10 client.15324 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2668421480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56487b2929a0 con 0x56487b2161a0 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f0897400640 1 -- 172.21.15.16:0/2668421480 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a80fa070 con 0x56487b2161a0 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f0897400640 1 -- 172.21.15.16:0/2668421480 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f08a400f040 con 0x56487b2161a0 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f08bf3e2740 10 client.15324 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f08bf3e2740 10 client.15324 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f08bf3e2740 20 client.15324 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x56487b297220 0x56487b295f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:33.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f08bf3e2740 4 client.15324 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2668421480 --> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] -- command(tid 0: {"prefix": "status"}) -- 0x56487b1893b0 con 0x56487b297220 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.942+0000 7f08b9a00640 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x56487b297220 0x56487b295f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08b9a00640 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x56487b297220 0x56487b295f30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f2598e5fe1ca88e server_cookie=5fd4e9f9a747f92a in_seq=0 out_seq=0 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f0897400640 10 client.15324 ms_handle_connect on v2:172.21.15.119:6836/477256695 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f0897400640 1 -- 172.21.15.16:0/2668421480 <== mds.1 v2:172.21.15.119:6836/477256695 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56487b1893b0 con 0x56487b297220 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f0897400640 10 client.15324 handle_command_reply: tid=0 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08bf3e2740 1 client.15324 shutdown 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2668421480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2161a0 msgr2=0x56487b28ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2161a0 0x56487b28ebc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x56487b211c90 tx=0x7f08a4009ca0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2668421480 shutdown_connections 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.119:6836/477256695,v1:172.21.15.119:6837/477256695] conn(0x56487b297220 0x56487b295f30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56487b21b310 0x56487b291110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56487b2161a0 0x56487b28ebc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08bf3e2740 1 --2- 172.21.15.16:0/2668421480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56487b2157b0 0x56487b2251e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f0897400640 0 client.15324 ms_handle_reset on v2:172.21.15.119:6836/477256695 2024-04-13T03:34:33.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.943+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2668421480 >> 172.21.15.16:0/2668421480 conn(0x56487b210d20 msgr2=0x56487b22b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2668421480 shutdown_connections 2024-04-13T03:34:33.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bf3e2740 1 -- 172.21.15.16:0/2668421480 wait complete. 2024-04-13T03:34:33.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bf3e2740 20 client.15324 trim_cache size 0 max 16384 2024-04-13T03:34:33.949 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:34:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 1, 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24530, 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 149.91750403399999, 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 197.21406234400001 2024-04-13T03:34:33.950 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:34:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 -- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f089c0a3c30 msgr2=0x7f089c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f089c0a3c30 0x7f089c0a60f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f08ac005fd0 tx=0x7f08ac00b560 comp rx=0 tx=0).stop 2024-04-13T03:34:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 -- 172.21.15.16:0/1495150387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 msgr2=0x7f08b419a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 0x7f08b419a3f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f08a8000c00 tx=0x7f08a8005dd0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 -- 172.21.15.16:0/1495150387 shutdown_connections 2024-04-13T03:34:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f089c0a3c30 0x7f089c0a60f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08b4105cb0 0x7f08b41a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b41052f0 0x7f08b419a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 --2- 172.21.15.16:0/1495150387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08b4104100 0x7f08b419a3f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.944+0000 7f08bae00640 1 -- 172.21.15.16:0/1495150387 >> 172.21.15.16:0/1495150387 conn(0x7f08b40ff670 msgr2=0x7f08b4108590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.945+0000 7f08bae00640 1 -- 172.21.15.16:0/1495150387 shutdown_connections 2024-04-13T03:34:33.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:33.945+0000 7f08bae00640 1 -- 172.21.15.16:0/1495150387 wait complete. 2024-04-13T03:34:33.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-13T03:34:34.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:33 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2607638959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:34:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.185+0000 7f823ee00640 1 -- 172.21.15.16:0/2313009570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8238107c40 msgr2=0x7f823810a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.185+0000 7f823ee00640 1 --2- 172.21.15.16:0/2313009570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8238107c40 0x7f823810a030 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8230009c90 tx=0x7f823002c0b0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.185+0000 7f823ee00640 1 -- 172.21.15.16:0/2313009570 shutdown_connections 2024-04-13T03:34:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.185+0000 7f823ee00640 1 --2- 172.21.15.16:0/2313009570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8238107c40 0x7f823810a030 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.185+0000 7f823ee00640 1 --2- 172.21.15.16:0/2313009570 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8238105310 0x7f8238107700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.185+0000 7f823ee00640 1 --2- 172.21.15.16:0/2313009570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82381029e0 0x7f8238104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.185+0000 7f823ee00640 1 -- 172.21.15.16:0/2313009570 >> 172.21.15.16:0/2313009570 conn(0x7f82380fc370 msgr2=0x7f82380fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.185+0000 7f823ee00640 1 -- 172.21.15.16:0/2313009570 shutdown_connections 2024-04-13T03:34:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.185+0000 7f823ee00640 1 -- 172.21.15.16:0/2313009570 wait complete. 2024-04-13T03:34:34.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.186+0000 7f823ee00640 1 Processor -- start 2024-04-13T03:34:34.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.186+0000 7f823ee00640 1 -- start start 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823ee00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82381029e0 0x7f823819a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823ee00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8238105310 0x7f823819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823ee00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8238107c40 0x7f82381a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823ee00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f823810dba0 con 0x7f82381029e0 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823ee00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f823810da20 con 0x7f8238105310 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823ee00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f823810dd20 con 0x7f8238107c40 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82381029e0 0x7f823819a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823d000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8238105310 0x7f823819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82381029e0 0x7f823819a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39116/0 (socket says 172.21.15.16:39116) 2024-04-13T03:34:34.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823da00640 1 -- 172.21.15.16:0/835665495 learned_addr learned my addr 172.21.15.16:0/835665495 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823e400640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8238107c40 0x7f82381a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823d000640 1 -- 172.21.15.16:0/835665495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8238107c40 msgr2=0x7f82381a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823d000640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8238107c40 0x7f82381a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823d000640 1 -- 172.21.15.16:0/835665495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82381029e0 msgr2=0x7f823819a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823d000640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82381029e0 0x7f823819a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.187+0000 7f823d000640 1 -- 172.21.15.16:0/835665495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82381a1ff0 con 0x7f8238105310 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.188+0000 7f823d000640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8238105310 0x7f823819a910 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f822800e9c0 tx=0x7f822800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.188+0000 7f8236000640 1 -- 172.21.15.16:0/835665495 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822800ccc0 con 0x7f8238105310 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.188+0000 7f8236000640 1 -- 172.21.15.16:0/835665495 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8228010910 con 0x7f8238105310 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.188+0000 7f823ee00640 1 -- 172.21.15.16:0/835665495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82381a22e0 con 0x7f8238105310 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.188+0000 7f8236000640 1 -- 172.21.15.16:0/835665495 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82280188c0 con 0x7f8238105310 2024-04-13T03:34:34.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.188+0000 7f823ee00640 1 -- 172.21.15.16:0/835665495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82381a2880 con 0x7f8238105310 2024-04-13T03:34:34.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.191+0000 7f8243436740 1 Processor -- start 2024-04-13T03:34:34.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.191+0000 7f8236000640 1 -- 172.21.15.16:0/835665495 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8228010ac0 con 0x7f8238105310 2024-04-13T03:34:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f8243436740 1 -- start start 2024-04-13T03:34:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f8243436740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5a98bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f8243436740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5a9dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f8243436740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aa07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f8243436740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e5d59fe990 con 0x55e5d5a9e310 2024-04-13T03:34:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f8243436740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e5d59a5530 con 0x55e5d5a991a0 2024-04-13T03:34:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f8243436740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e5d59fa950 con 0x55e5d5a987b0 2024-04-13T03:34:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f823da00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5a9dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f823e400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5a98bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f823da00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5a9dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56286/0 (socket says 172.21.15.16:56286) 2024-04-13T03:34:34.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f823da00640 1 -- 172.21.15.16:0/3537031880 learned_addr learned my addr 172.21.15.16:0/3537031880 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.192+0000 7f823d000640 1 --2- 172.21.15.16:0/3537031880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aa07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.193+0000 7f823e400640 1 -- 172.21.15.16:0/3537031880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 msgr2=0x55e5d5a9dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.193+0000 7f823e400640 1 --2- 172.21.15.16:0/3537031880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5a9dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.193+0000 7f823e400640 1 -- 172.21.15.16:0/3537031880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 msgr2=0x55e5d5aa07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.193+0000 7f823e400640 1 --2- 172.21.15.16:0/3537031880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aa07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.193+0000 7f823e400640 1 -- 172.21.15.16:0/3537031880 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5d57112d0 con 0x55e5d5a987b0 2024-04-13T03:34:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.193+0000 7f823da00640 1 --2- 172.21.15.16:0/3537031880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5a9dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:34.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.193+0000 7f823e400640 1 --2- 172.21.15.16:0/3537031880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5a98bd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f82300011e0 tx=0x7f8230009970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=252bf6a53dec641f server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:34.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.193+0000 7f8236000640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82200a3c80 0x7f82200a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f8226a00640 1 -- 172.21.15.16:0/3537031880 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230039040 con 0x55e5d5a987b0 2024-04-13T03:34:34.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f8226a00640 1 -- 172.21.15.16:0/3537031880 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8230004a90 con 0x55e5d5a987b0 2024-04-13T03:34:34.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f8226a00640 1 -- 172.21.15.16:0/3537031880 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230004030 con 0x55e5d5a987b0 2024-04-13T03:34:34.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f8236000640 1 -- 172.21.15.16:0/835665495 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f82280d0b10 con 0x7f8238105310 2024-04-13T03:34:34.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f823da00640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82200a3c80 0x7f82200a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f8243436740 1 -- 172.21.15.16:0/3537031880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 msgr2=0x55e5d5a98bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f8243436740 1 --2- 172.21.15.16:0/3537031880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5a98bd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f82300011e0 tx=0x7f8230009970 comp rx=0 tx=0).stop 2024-04-13T03:34:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f8243436740 1 -- 172.21.15.16:0/3537031880 shutdown_connections 2024-04-13T03:34:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f8243436740 1 --2- 172.21.15.16:0/3537031880 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aa07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.194+0000 7f8243436740 1 --2- 172.21.15.16:0/3537031880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5a9dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.195+0000 7f8243436740 1 --2- 172.21.15.16:0/3537031880 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5a98bd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.195+0000 7f8243436740 1 -- 172.21.15.16:0/3537031880 >> 172.21.15.16:0/3537031880 conn(0x55e5d5a93d20 msgr2=0x55e5d5a94120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.195+0000 7f823da00640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82200a3c80 0x7f82200a6140 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55e5d5a995a0 tx=0x7f822c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.195+0000 7f8243436740 1 -- 172.21.15.16:0/3537031880 shutdown_connections 2024-04-13T03:34:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.195+0000 7f8243436740 1 -- 172.21.15.16:0/3537031880 wait complete. 2024-04-13T03:34:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.195+0000 7f8243436740 1 -- start start 2024-04-13T03:34:34.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.195+0000 7f8243436740 1 Processor -- start 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f8243436740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f8243436740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aa3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f8243436740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aaa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f8243436740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e5d5aa1260 con 0x55e5d5a9e310 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f8243436740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e5d59b88f0 con 0x55e5d5a991a0 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f8243436740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e5d5aa13e0 con 0x55e5d5a987b0 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f8243436740 10 client.0 _ll_register_callbacks cb 0x55e5d59b82e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f823da00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f823da00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54238/0 (socket says 172.21.15.16:54238) 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f823da00640 1 -- 172.21.15.16:0/3753950717 learned_addr learned my addr 172.21.15.16:0/3753950717 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:34.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f823e400640 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aaa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f823d000640 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aa3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.196+0000 7f823da00640 1 -- 172.21.15.16:0/3753950717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 msgr2=0x55e5d5aa3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f823da00640 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aa3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f823da00640 1 -- 172.21.15.16:0/3753950717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 msgr2=0x55e5d5aaa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f823da00640 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aaa250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f823da00640 1 -- 172.21.15.16:0/3753950717 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5d57db6b0 con 0x55e5d5a987b0 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f823e400640 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aaa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f823da00640 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa2cd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f822c009590 tx=0x7f822c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53e72f018de2b798 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f821b400640 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f821b400640 1 -- 172.21.15.16:0/3753950717 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c00f750 con 0x55e5d5a987b0 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f821b400640 1 -- 172.21.15.16:0/3753950717 <== mon.1 v2:172.21.15.188:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f822c00f8f0 con 0x55e5d5a987b0 2024-04-13T03:34:34.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f8243436740 1 -- 172.21.15.16:0/3753950717 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8230004a90 con 0x55e5d5a987b0 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f821b400640 1 -- 172.21.15.16:0/3753950717 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f822c00faf0 con 0x55e5d5a987b0 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.197+0000 7f8243436740 10 client.25423 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f8243436740 1 -- 172.21.15.16:0/3753950717 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5d5aad690 con 0x55e5d5a987b0 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f821b400640 1 -- 172.21.15.16:0/3753950717 <== mon.1 v2:172.21.15.188:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8230004a90 con 0x55e5d5a987b0 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f821b400640 1 -- 172.21.15.16:0/3753950717 <== mon.1 v2:172.21.15.188:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f822c027070 con 0x55e5d5a987b0 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f8243436740 10 client.25423 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f8243436740 10 client.25423 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f8243436740 20 client.25423 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f8243436740 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55e5d5aafd70 0x55e5d5ab11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f8243436740 4 client.25423 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f8243436740 1 -- 172.21.15.16:0/3753950717 --> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5d59a4b40 con 0x55e5d5aafd70 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.198+0000 7f823d000640 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55e5d5aafd70 0x55e5d5ab11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.199+0000 7f823d000640 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55e5d5aafd70 0x55e5d5ab11e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97e645e03e547bc0 server_cookie=9599e409fe2f6de3 in_seq=0 out_seq=0 2024-04-13T03:34:34.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.199+0000 7f821b400640 10 client.25423 ms_handle_connect on v2:172.21.15.119:6838/3722020849 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f821b400640 1 -- 172.21.15.16:0/3753950717 <== mds.0 v2:172.21.15.119:6838/3722020849 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5d59a4b40 con 0x55e5d5aafd70 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f821b400640 10 client.25423 handle_command_reply: tid=0 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 client.25423 shutdown 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 -- 172.21.15.16:0/3753950717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 msgr2=0x55e5d5aa2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa2cd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f822c009590 tx=0x7f822c004730 comp rx=0 tx=0).stop 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 -- 172.21.15.16:0/3753950717 shutdown_connections 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55e5d5aafd70 0x55e5d5ab11e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aaa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aa3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f821b400640 0 client.25423 ms_handle_reset on v2:172.21.15.119:6838/3722020849 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 --2- 172.21.15.16:0/3753950717 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa2cd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 -- 172.21.15.16:0/3753950717 >> 172.21.15.16:0/3753950717 conn(0x55e5d5a93d20 msgr2=0x55e5d5a92f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:34.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.200+0000 7f8243436740 1 -- 172.21.15.16:0/3753950717 shutdown_connections 2024-04-13T03:34:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.201+0000 7f8243436740 1 -- 172.21.15.16:0/3753950717 wait complete. 2024-04-13T03:34:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.201+0000 7f8243436740 20 client.25423 trim_cache size 0 max 16384 2024-04-13T03:34:34.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.220+0000 7f8243436740 1 Processor -- start 2024-04-13T03:34:34.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f8243436740 1 -- start start 2024-04-13T03:34:34.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f8243436740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f8243436740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aaa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f8243436740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aaeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f8243436740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e5d5aaf520 con 0x55e5d5a9e310 2024-04-13T03:34:34.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f8243436740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e5d5aafbf0 con 0x55e5d5a991a0 2024-04-13T03:34:34.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f8243436740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e5d5ab02c0 con 0x55e5d5a987b0 2024-04-13T03:34:34.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f823da00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f823d000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aaa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f823da00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54250/0 (socket says 172.21.15.16:54250) 2024-04-13T03:34:34.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f823da00640 1 -- 172.21.15.16:0/3723358148 learned_addr learned my addr 172.21.15.16:0/3723358148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:34.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.221+0000 7f823d000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aaa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56296/0 (socket says 172.21.15.16:56296) 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f823e400640 1 --2- 172.21.15.16:0/3723358148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aaeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f823d000640 1 -- 172.21.15.16:0/3723358148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 msgr2=0x55e5d5aa0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f823d000640 1 --2- 172.21.15.16:0/3723358148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f823d000640 1 -- 172.21.15.16:0/3723358148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 msgr2=0x55e5d5aaeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f823d000640 1 --2- 172.21.15.16:0/3723358148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aaeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f823d000640 1 -- 172.21.15.16:0/3723358148 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5d5a9d9b0 con 0x55e5d5a991a0 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f823d000640 1 --2- 172.21.15.16:0/3723358148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aaa1e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f822809c770 tx=0x7f82280ce0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=380543d5633c98fa server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f8226000640 1 -- 172.21.15.16:0/3723358148 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822809c3d0 con 0x55e5d5a991a0 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f8226000640 1 -- 172.21.15.16:0/3723358148 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82280d9040 con 0x55e5d5a991a0 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.222+0000 7f8226000640 1 -- 172.21.15.16:0/3723358148 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82280990e0 con 0x55e5d5a991a0 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 -- 172.21.15.16:0/3723358148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 msgr2=0x55e5d5aaa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 --2- 172.21.15.16:0/3723358148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aaa1e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f822809c770 tx=0x7f82280ce0b0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 -- 172.21.15.16:0/3723358148 shutdown_connections 2024-04-13T03:34:34.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 --2- 172.21.15.16:0/3723358148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a9e310 0x55e5d5aaeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 --2- 172.21.15.16:0/3723358148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a991a0 0x55e5d5aaa1e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 --2- 172.21.15.16:0/3723358148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 -- 172.21.15.16:0/3723358148 >> 172.21.15.16:0/3723358148 conn(0x55e5d5a93d20 msgr2=0x55e5d5aa17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:34.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 -- 172.21.15.16:0/3723358148 shutdown_connections 2024-04-13T03:34:34.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 -- 172.21.15.16:0/3723358148 wait complete. 2024-04-13T03:34:34.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.223+0000 7f8243436740 1 -- start start 2024-04-13T03:34:34.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.224+0000 7f8243436740 1 Processor -- start 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.224+0000 7f8243436740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.224+0000 7f8243436740 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a991a0 0x55e5d5b11d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.224+0000 7f8243436740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a9e310 0x55e5d5b14250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.224+0000 7f8243436740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x55e5d5ab1060 con 0x55e5d5a987b0 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.224+0000 7f8243436740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e5d5ab0990 con 0x55e5d5a9e310 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.224+0000 7f8243436740 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x55e5d5ab1730 con 0x55e5d5a991a0 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.224+0000 7f8243436740 10 client.0 _ll_register_callbacks cb 0x7f822c0043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.224+0000 7f823da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39168/0 (socket says 172.21.15.16:39168) 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823da00640 1 -- 172.21.15.16:0/4246341959 learned_addr learned my addr 172.21.15.16:0/4246341959 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823e400640 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a9e310 0x55e5d5b14250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823d000640 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a991a0 0x55e5d5b11d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823e400640 1 -- 172.21.15.16:0/4246341959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a991a0 msgr2=0x55e5d5b11d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823e400640 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a991a0 0x55e5d5b11d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823e400640 1 -- 172.21.15.16:0/4246341959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a987b0 msgr2=0x55e5d5aa8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823e400640 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823e400640 1 -- 172.21.15.16:0/4246341959 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5d5b14790 con 0x55e5d5a9e310 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f823e400640 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a9e310 0x55e5d5b14250 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8230037040 tx=0x7f8230005aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c99c305333e3d048 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.225+0000 7f821b400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f821b400640 1 -- 172.21.15.16:0/4246341959 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230039040 con 0x55e5d5a9e310 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f8243436740 1 -- 172.21.15.16:0/4246341959 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82280d9040 con 0x55e5d5a9e310 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f821b400640 1 -- 172.21.15.16:0/4246341959 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8230003bc0 con 0x55e5d5a9e310 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f821b400640 1 -- 172.21.15.16:0/4246341959 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8230005b90 con 0x55e5d5a9e310 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f8243436740 10 client.25448 fetch_fsmap learned FSMap version 26 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f8243436740 1 -- 172.21.15.16:0/4246341959 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5d5b15990 con 0x55e5d5a9e310 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f821b400640 1 -- 172.21.15.16:0/4246341959 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82280d9040 con 0x55e5d5a9e310 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f821b400640 1 -- 172.21.15.16:0/4246341959 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 26) ==== 8362+0+0 (secure 0 0 0) 0x7f823003f5a0 con 0x55e5d5a9e310 2024-04-13T03:34:34.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f8243436740 10 client.25448 fetch_fsmap finished waiting for FSMap version 26 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f8243436740 10 client.25448 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f8243436740 20 client.25448 populate_metadata read hostname 'smithi016' 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f8243436740 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55e5d5b1a140 0x55e5d5b18e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f8243436740 4 client.25448 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.226+0000 7f8243436740 1 -- 172.21.15.16:0/4246341959 --> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] -- command(tid 0: {"prefix": "status"}) -- 0x55e5d5a0c3b0 con 0x55e5d5b1a140 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.227+0000 7f823da00640 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55e5d5b1a140 0x55e5d5b18e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.227+0000 7f823da00640 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55e5d5b1a140 0x55e5d5b18e50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb284dce997aec8a server_cookie=3b5bf88338c1e0b3 in_seq=0 out_seq=0 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.227+0000 7f821b400640 10 client.25448 ms_handle_connect on v2:172.21.15.119:6838/3722020849 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.227+0000 7f821b400640 1 -- 172.21.15.16:0/4246341959 <== mds.0 v2:172.21.15.119:6838/3722020849 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e5d5a0c3b0 con 0x55e5d5b1a140 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.227+0000 7f821b400640 10 client.25448 handle_command_reply: tid=0 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.227+0000 7f8243436740 1 client.25448 shutdown 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 -- 172.21.15.16:0/4246341959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a9e310 msgr2=0x55e5d5b14250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a9e310 0x55e5d5b14250 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8230037040 tx=0x7f8230005aa0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 -- 172.21.15.16:0/4246341959 shutdown_connections 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.119:6838/3722020849,v1:172.21.15.119:6839/3722020849] conn(0x55e5d5b1a140 0x55e5d5b18e50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e5d5a9e310 0x55e5d5b14250 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55e5d5a991a0 0x55e5d5b11d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 --2- 172.21.15.16:0/4246341959 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x55e5d5a987b0 0x55e5d5aa8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f821b400640 0 client.25448 ms_handle_reset on v2:172.21.15.119:6838/3722020849 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 -- 172.21.15.16:0/4246341959 >> 172.21.15.16:0/4246341959 conn(0x55e5d5a93d20 msgr2=0x55e5d5aaeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 -- 172.21.15.16:0/4246341959 shutdown_connections 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 1 -- 172.21.15.16:0/4246341959 wait complete. 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.228+0000 7f8243436740 20 client.25448 trim_cache size 0 max 16384 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:34:34.232 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "e6bf0314-f944-11ee-bc8d-c7b262605968", 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 24587, 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 176.67532902799999, 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 26, 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 94, 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 90, 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 183.05107280600001 2024-04-13T03:34:34.233 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:34:34.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 -- 172.21.15.16:0/835665495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82200a3c80 msgr2=0x7f82200a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82200a3c80 0x7f82200a6140 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55e5d5a995a0 tx=0x7f822c005f90 comp rx=0 tx=0).stop 2024-04-13T03:34:34.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 -- 172.21.15.16:0/835665495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8238105310 msgr2=0x7f823819a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8238105310 0x7f823819a910 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f822800e9c0 tx=0x7f822800ee80 comp rx=0 tx=0).stop 2024-04-13T03:34:34.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 -- 172.21.15.16:0/835665495 shutdown_connections 2024-04-13T03:34:34.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f82200a3c80 0x7f82200a6140 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8238107c40 0x7f82381a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8238105310 0x7f823819a910 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 --2- 172.21.15.16:0/835665495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82381029e0 0x7f823819a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 -- 172.21.15.16:0/835665495 >> 172.21.15.16:0/835665495 conn(0x7f82380fc370 msgr2=0x7f82381020b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:34.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 -- 172.21.15.16:0/835665495 shutdown_connections 2024-04-13T03:34:34.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.229+0000 7f823ee00640 1 -- 172.21.15.16:0/835665495 wait complete. 2024-04-13T03:34:34.248 INFO:teuthology.run_tasks:Running task sequential... 2024-04-13T03:34:34.260 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-13T03:34:34.260 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-13T03:34:34.260 INFO:teuthology.task.print:Enabling mgr modules 2024-04-13T03:34:34.260 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-13T03:34:34.261 INFO:teuthology.task.exec:Executing custom commands... 2024-04-13T03:34:34.261 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T03:34:34.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-13T03:34:34.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.483+0000 7fb792c00640 1 -- 172.21.15.16:0/3364932044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb78c077bc0 msgr2=0x7fb78c078020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.483+0000 7fb792c00640 1 --2- 172.21.15.16:0/3364932044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb78c077bc0 0x7fb78c078020 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb778009900 tx=0x7fb77802bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.483+0000 7fb792c00640 1 -- 172.21.15.16:0/3364932044 shutdown_connections 2024-04-13T03:34:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.483+0000 7fb792c00640 1 --2- 172.21.15.16:0/3364932044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb78c078560 0x7fb78c078e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.483+0000 7fb792c00640 1 --2- 172.21.15.16:0/3364932044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb78c077bc0 0x7fb78c078020 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.483+0000 7fb792c00640 1 --2- 172.21.15.16:0/3364932044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c076980 0x7fb78c076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.483+0000 7fb792c00640 1 -- 172.21.15.16:0/3364932044 >> 172.21.15.16:0/3364932044 conn(0x7fb78c0ff5f0 msgr2=0x7fb78c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.483+0000 7fb792c00640 1 -- 172.21.15.16:0/3364932044 shutdown_connections 2024-04-13T03:34:34.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.483+0000 7fb792c00640 1 -- 172.21.15.16:0/3364932044 wait complete. 2024-04-13T03:34:34.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.484+0000 7fb792c00640 1 Processor -- start 2024-04-13T03:34:34.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.485+0000 7fb792c00640 1 -- start start 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.485+0000 7fb792c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb78c076980 0x7fb78c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.485+0000 7fb792c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb78c077bc0 0x7fb78c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.485+0000 7fb792c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c078560 0x7fb78c1a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.485+0000 7fb792c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c114110 con 0x7fb78c078560 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.485+0000 7fb792c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c113f90 con 0x7fb78c077bc0 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.485+0000 7fb792c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c114290 con 0x7fb78c076980 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.485+0000 7fb791800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb78c076980 0x7fb78c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb791800640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb78c076980 0x7fb78c19e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54276/0 (socket says 172.21.15.16:54276) 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb791800640 1 -- 172.21.15.16:0/9010321 learned_addr learned my addr 172.21.15.16:0/9010321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb790e00640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb78c077bc0 0x7fb78c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb792200640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c078560 0x7fb78c1a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb791800640 1 -- 172.21.15.16:0/9010321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb78c077bc0 msgr2=0x7fb78c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb791800640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb78c077bc0 0x7fb78c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb791800640 1 -- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c078560 msgr2=0x7fb78c1a5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb791800640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c078560 0x7fb78c1a5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb791800640 1 -- 172.21.15.16:0/9010321 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78c1a6350 con 0x7fb78c076980 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.486+0000 7fb791800640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb78c076980 0x7fb78c19e730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb78000d950 tx=0x7fb78000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.487+0000 7fb78a000640 1 -- 172.21.15.16:0/9010321 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb780017030 con 0x7fb78c076980 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.487+0000 7fb78a000640 1 -- 172.21.15.16:0/9010321 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb780004020 con 0x7fb78c076980 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.487+0000 7fb78a000640 1 -- 172.21.15.16:0/9010321 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7800047a0 con 0x7fb78c076980 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.487+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb78c1a6640 con 0x7fb78c076980 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.487+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb78c1a6b10 con 0x7fb78c076980 2024-04-13T03:34:34.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.488+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb78c1109f0 con 0x7fb78c076980 2024-04-13T03:34:34.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.491+0000 7fb78a000640 1 -- 172.21.15.16:0/9010321 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb7800041d0 con 0x7fb78c076980 2024-04-13T03:34:34.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.492+0000 7fb78a000640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb7740a3ba0 0x7fb7740a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.493+0000 7fb78a000640 1 -- 172.21.15.16:0/9010321 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb7800d01d0 con 0x7fb78c076980 2024-04-13T03:34:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.493+0000 7fb790e00640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb7740a3ba0 0x7fb7740a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:34.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.494+0000 7fb790e00640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb7740a3ba0 0x7fb7740a6060 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb77802c7b0 tx=0x7fb778005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:34.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.497+0000 7fb78a000640 1 -- 172.21.15.16:0/9010321 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb78009c7e0 con 0x7fb78c076980 2024-04-13T03:34:34.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:34.763+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fb78c062bf0 con 0x7fb78c076980 2024-04-13T03:34:35.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:34 smithi119 ceph-mon[35835]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:35.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:34 smithi016 ceph-mon[31772]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:34 smithi188 ceph-mon[36398]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-13T03:34:35.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.653+0000 7fb78a000640 1 -- 172.21.15.16:0/9010321 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fb780098750 con 0x7fb78c076980 2024-04-13T03:34:35.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.654+0000 7fb78a000640 1 -- 172.21.15.16:0/9010321 <== mon.1 v2:172.21.15.188:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fb7800786a0 con 0x7fb78c076980 2024-04-13T03:34:35.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.659+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb7740a3ba0 msgr2=0x7fb7740a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:35.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.659+0000 7fb792c00640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb7740a3ba0 0x7fb7740a6060 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb77802c7b0 tx=0x7fb778005fd0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.659+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb78c076980 msgr2=0x7fb78c19e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:35.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.659+0000 7fb792c00640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb78c076980 0x7fb78c19e730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb78000d950 tx=0x7fb78000de10 comp rx=0 tx=0).stop 2024-04-13T03:34:35.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.660+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 shutdown_connections 2024-04-13T03:34:35.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.660+0000 7fb792c00640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7fb7740a3ba0 0x7fb7740a6060 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.660+0000 7fb792c00640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c078560 0x7fb78c1a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.660+0000 7fb792c00640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb78c077bc0 0x7fb78c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.660+0000 7fb792c00640 1 --2- 172.21.15.16:0/9010321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb78c076980 0x7fb78c19e730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.660+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 >> 172.21.15.16:0/9010321 conn(0x7fb78c0ff5f0 msgr2=0x7fb78c1010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:35.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.660+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 shutdown_connections 2024-04-13T03:34:35.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.660+0000 7fb792c00640 1 -- 172.21.15.16:0/9010321 wait complete. 2024-04-13T03:34:35.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-13T03:34:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.894+0000 7f1c15200640 1 -- 172.21.15.16:0/653950936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c1010e4b0 msgr2=0x7f1c101108a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.894+0000 7f1c15200640 1 --2- 172.21.15.16:0/653950936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c1010e4b0 0x7f1c101108a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009cd0 tx=0x7f1c0802c0f0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.895+0000 7f1c15200640 1 -- 172.21.15.16:0/653950936 shutdown_connections 2024-04-13T03:34:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.895+0000 7f1c15200640 1 --2- 172.21.15.16:0/653950936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c1010e4b0 0x7f1c101108a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.895+0000 7f1c15200640 1 --2- 172.21.15.16:0/653950936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c10074a20 0x7f1c10074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.895+0000 7f1c15200640 1 --2- 172.21.15.16:0/653950936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c10076000 0x7f1c10074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.895+0000 7f1c15200640 1 -- 172.21.15.16:0/653950936 >> 172.21.15.16:0/653950936 conn(0x7f1c100fd450 msgr2=0x7f1c100ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:35.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.895+0000 7f1c15200640 1 -- 172.21.15.16:0/653950936 shutdown_connections 2024-04-13T03:34:35.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.895+0000 7f1c15200640 1 -- 172.21.15.16:0/653950936 wait complete. 2024-04-13T03:34:35.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.896+0000 7f1c15200640 1 Processor -- start 2024-04-13T03:34:35.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.896+0000 7f1c15200640 1 -- start start 2024-04-13T03:34:35.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.896+0000 7f1c15200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c10074a20 0x7f1c1019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:35.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.896+0000 7f1c15200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c10076000 0x7f1c1019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:35.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.896+0000 7f1c15200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1010e4b0 0x7f1c101a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:35.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.896+0000 7f1c15200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c10114100 con 0x7f1c1010e4b0 2024-04-13T03:34:35.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.896+0000 7f1c15200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1c10113f80 con 0x7f1c10074a20 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.896+0000 7f1c15200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1c10114280 con 0x7f1c10076000 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c10074a20 0x7f1c1019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c10074a20 0x7f1c1019e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56356/0 (socket says 172.21.15.16:56356) 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0f400640 1 -- 172.21.15.16:0/3824490444 learned_addr learned my addr 172.21.15.16:0/3824490444 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0ea00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c10076000 0x7f1c1019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0fe00640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1010e4b0 0x7f1c101a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0f400640 1 -- 172.21.15.16:0/3824490444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c10076000 msgr2=0x7f1c1019ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0f400640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c10076000 0x7f1c1019ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0f400640 1 -- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1010e4b0 msgr2=0x7f1c101a5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0f400640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1010e4b0 0x7f1c101a5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0f400640 1 -- 172.21.15.16:0/3824490444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c101a62e0 con 0x7f1c10074a20 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.897+0000 7f1c0ea00640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c10076000 0x7f1c1019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:35.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.898+0000 7f1c0f400640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c10074a20 0x7f1c1019e6c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000e9f0 tx=0x7f1c0000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.898+0000 7f1c07e00640 1 -- 172.21.15.16:0/3824490444 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000cd80 con 0x7f1c10074a20 2024-04-13T03:34:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.898+0000 7f1c07e00640 1 -- 172.21.15.16:0/3824490444 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c00010910 con 0x7f1c10074a20 2024-04-13T03:34:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.898+0000 7f1c07e00640 1 -- 172.21.15.16:0/3824490444 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00018990 con 0x7f1c10074a20 2024-04-13T03:34:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.898+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c101a65d0 con 0x7f1c10074a20 2024-04-13T03:34:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.898+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c101a6b70 con 0x7f1c10074a20 2024-04-13T03:34:35.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.900+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c10104570 con 0x7f1c10074a20 2024-04-13T03:34:35.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.901+0000 7f1c07e00640 1 -- 172.21.15.16:0/3824490444 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f1c00010ac0 con 0x7f1c10074a20 2024-04-13T03:34:35.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.903+0000 7f1c07e00640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1bf40a3c70 0x7f1bf40a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:35.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.903+0000 7f1c0ea00640 1 -- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1bf40a3c70 msgr2=0x7f1bf40a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1721562410 2024-04-13T03:34:35.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.903+0000 7f1c07e00640 1 -- 172.21.15.16:0/3824490444 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1c000d0b00 con 0x7f1c10074a20 2024-04-13T03:34:35.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.903+0000 7f1c0ea00640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1bf40a3c70 0x7f1bf40a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:34:35.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:35.907+0000 7f1c07e00640 1 -- 172.21.15.16:0/3824490444 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c0009ca60 con 0x7f1c10074a20 2024-04-13T03:34:36.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:35 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/9010321' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-13T03:34:36.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:35 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-13T03:34:36.072 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:35 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: ignoring --setuser ceph since I am not root 2024-04-13T03:34:36.072 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:35 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: ignoring --setgroup ceph since I am not root 2024-04-13T03:34:36.072 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:35 smithi119 ceph-mgr[37237]: -- 172.21.15.119:0/2823879648 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b816e2e5a0 con 0x55b816e02c00 2024-04-13T03:34:36.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.104+0000 7f1c0ea00640 1 -- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1bf40a3c70 msgr2=0x7f1bf40a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1721562410 2024-04-13T03:34:36.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.104+0000 7f1c0ea00640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1bf40a3c70 0x7f1bf40a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-13T03:34:36.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:35 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/9010321' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-13T03:34:36.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:35 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-13T03:34:36.105 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:35 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: ignoring --setuser ceph since I am not root 2024-04-13T03:34:36.105 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:35 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: ignoring --setgroup ceph since I am not root 2024-04-13T03:34:36.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:35 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/9010321' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-13T03:34:36.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:35 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-13T03:34:36.123 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:35 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: ignoring --setuser ceph since I am not root 2024-04-13T03:34:36.123 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:35 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: ignoring --setgroup ceph since I am not root 2024-04-13T03:34:36.123 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:35 smithi188 ceph-mgr[38266]: -- 172.21.15.188:0/2825740937 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556ccff385a0 con 0x556ccff0cc00 2024-04-13T03:34:36.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.168+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f1c10062bc0 con 0x7f1c10074a20 2024-04-13T03:34:36.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.176+0000 7f1c07e00640 1 -- 172.21.15.16:0/3824490444 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f1c000a0d60 con 0x7f1c10074a20 2024-04-13T03:34:36.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1bf40a3c70 msgr2=0x7f1bf40a6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:34:36.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1bf40a3c70 0x7f1bf40a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c10074a20 msgr2=0x7f1c1019e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:36.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c10074a20 0x7f1c1019e6c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000e9f0 tx=0x7f1c0000eeb0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 shutdown_connections 2024-04-13T03:34:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f1bf40a3c70 0x7f1bf40a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c1010e4b0 0x7f1c101a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1c10076000 0x7f1c1019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 --2- 172.21.15.16:0/3824490444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c10074a20 0x7f1c1019e6c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 >> 172.21.15.16:0/3824490444 conn(0x7f1c100fd450 msgr2=0x7f1c10101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 shutdown_connections 2024-04-13T03:34:36.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.179+0000 7f1c15200640 1 -- 172.21.15.16:0/3824490444 wait complete. 2024-04-13T03:34:36.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-13T03:34:36.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.396+0000 7f24f4400640 1 -- 172.21.15.16:0/2073952699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 msgr2=0x7f24ec104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.396+0000 7f24f4400640 1 --2- 172.21.15.16:0/2073952699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 0x7f24ec104e90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009900 tx=0x7f24e002bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.396+0000 7f24f4400640 1 -- 172.21.15.16:0/2073952699 shutdown_connections 2024-04-13T03:34:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.396+0000 7f24f4400640 1 --2- 172.21.15.16:0/2073952699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec107d00 0x7f24ec10a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.396+0000 7f24f4400640 1 --2- 172.21.15.16:0/2073952699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec1053d0 0x7f24ec1077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.396+0000 7f24f4400640 1 --2- 172.21.15.16:0/2073952699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 0x7f24ec104e90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.396+0000 7f24f4400640 1 -- 172.21.15.16:0/2073952699 >> 172.21.15.16:0/2073952699 conn(0x7f24ec0fc490 msgr2=0x7f24ec0fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.396+0000 7f24f4400640 1 -- 172.21.15.16:0/2073952699 shutdown_connections 2024-04-13T03:34:36.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.396+0000 7f24f4400640 1 -- 172.21.15.16:0/2073952699 wait complete. 2024-04-13T03:34:36.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.397+0000 7f24f4400640 1 Processor -- start 2024-04-13T03:34:36.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.397+0000 7f24f4400640 1 -- start start 2024-04-13T03:34:36.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f4400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 0x7f24ec19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f4400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec1053d0 0x7f24ec19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f4400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec107d00 0x7f24ec1a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f4400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec10dd00 con 0x7f24ec107d00 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f4400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec10db80 con 0x7f24ec102aa0 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f4400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec10de80 con 0x7f24ec1053d0 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 0x7f24ec19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec1053d0 0x7f24ec19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec1053d0 0x7f24ec19a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:54308/0 (socket says 172.21.15.16:54308) 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 0x7f24ec19a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:56376/0 (socket says 172.21.15.16:56376) 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1000640 1 -- 172.21.15.16:0/1790879226 learned_addr learned my addr 172.21.15.16:0/1790879226 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f2400640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec107d00 0x7f24ec1a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1a00640 1 -- 172.21.15.16:0/1790879226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec1053d0 msgr2=0x7f24ec19a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:36.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1a00640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec1053d0 0x7f24ec19a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1a00640 1 -- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec107d00 msgr2=0x7f24ec1a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1a00640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec107d00 0x7f24ec1a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.398+0000 7f24f1a00640 1 -- 172.21.15.16:0/1790879226 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24ec1a2080 con 0x7f24ec102aa0 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.399+0000 7f24f1a00640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 0x7f24ec19a460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f24e0036040 tx=0x7f24e002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.399+0000 7f24ea000640 1 -- 172.21.15.16:0/1790879226 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0043020 con 0x7f24ec102aa0 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.399+0000 7f24ea000640 1 -- 172.21.15.16:0/1790879226 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e00054e0 con 0x7f24ec102aa0 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.399+0000 7f24ea000640 1 -- 172.21.15.16:0/1790879226 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0003c30 con 0x7f24ec102aa0 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.399+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24ec1a2310 con 0x7f24ec102aa0 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.399+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24ec1a2880 con 0x7f24ec102aa0 2024-04-13T03:34:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.400+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24ec109850 con 0x7f24ec102aa0 2024-04-13T03:34:36.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.402+0000 7f24ea000640 1 -- 172.21.15.16:0/1790879226 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f24e0046020 con 0x7f24ec102aa0 2024-04-13T03:34:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.403+0000 7f24ea000640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d40a3d20 0x7f24d40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.403+0000 7f24ea000640 1 -- 172.21.15.16:0/1790879226 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f24e00f27b0 con 0x7f24ec102aa0 2024-04-13T03:34:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.403+0000 7f24f1000640 1 -- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d40a3d20 msgr2=0x7f24d40a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1721562410 2024-04-13T03:34:36.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.403+0000 7f24f1000640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d40a3d20 0x7f24d40a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:34:36.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.407+0000 7f24ea000640 1 -- 172.21.15.16:0/1790879226 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24e004d020 con 0x7f24ec102aa0 2024-04-13T03:34:36.571 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:36 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:36.212+0000 7f6bfb6cd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T03:34:36.571 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:36 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:36.304+0000 7f6bfb6cd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T03:34:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.604+0000 7f24f1000640 1 -- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d40a3d20 msgr2=0x7f24d40a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1721562410 2024-04-13T03:34:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.604+0000 7f24f1000640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d40a3d20 0x7f24d40a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-13T03:34:36.622 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:36 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:36.212+0000 7f1ee7009200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T03:34:36.622 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:36 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:36.304+0000 7f1ee7009200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T03:34:36.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.700+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f24ec02d5e0 con 0x7f24ec102aa0 2024-04-13T03:34:36.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.708+0000 7f24ea000640 1 -- 172.21.15.16:0/1790879226 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f24e00bed60 con 0x7f24ec102aa0 2024-04-13T03:34:36.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d40a3d20 msgr2=0x7f24d40a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:34:36.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d40a3d20 0x7f24d40a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 msgr2=0x7f24ec19a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 0x7f24ec19a460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f24e0036040 tx=0x7f24e002c650 comp rx=0 tx=0).stop 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 shutdown_connections 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f24d40a3d20 0x7f24d40a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24ec107d00 0x7f24ec1a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24ec1053d0 0x7f24ec19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 --2- 172.21.15.16:0/1790879226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec102aa0 0x7f24ec19a460 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 >> 172.21.15.16:0/1790879226 conn(0x7f24ec0fc490 msgr2=0x7f24ec102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 shutdown_connections 2024-04-13T03:34:36.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:36.711+0000 7f24f4400640 1 -- 172.21.15.16:0/1790879226 wait complete. 2024-04-13T03:34:36.728 INFO:teuthology.run_tasks:Running task kclient... 2024-04-13T03:34:36.742 INFO:tasks.kclient:Mounting kernel clients... 2024-04-13T03:34:36.742 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-13T03:34:36.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:36.742 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-13T03:34:36.792 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-13T03:34:36.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:36.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-13T03:34:36.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:36.880 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-13T03:34:36.905 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:36.651+0000 7fee4853d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-13T03:34:36.905 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:36.786+0000 7fee4853d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-13T03:34:36.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:36 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-13T03:34:36.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:36 smithi016 ceph-mon[31772]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-13T03:34:36.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:36 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' 2024-04-13T03:34:36.944 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:36 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-13T03:34:36.944 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:36 smithi188 ceph-mon[36398]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-13T03:34:36.944 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:36 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' 2024-04-13T03:34:36.947 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:36 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-13T03:34:36.947 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:36 smithi119 ceph-mon[35835]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-13T03:34:36.947 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:36 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' 2024-04-13T03:34:36.998 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-13T03:34:36.998 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-13T03:34:36.998 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-13T03:34:36.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph fs ls 2024-04-13T03:34:37.274 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:36 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:36.944+0000 7f1ee7009200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T03:34:37.274 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:37.036+0000 7f1ee7009200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T03:34:37.274 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:37.178+0000 7f1ee7009200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T03:34:37.274 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:37.273+0000 7f1ee7009200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T03:34:37.276 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:36 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:36.946+0000 7f6bfb6cd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T03:34:37.276 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:37.039+0000 7f6bfb6cd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T03:34:37.277 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:37.181+0000 7f6bfb6cd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T03:34:37.277 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:37.275+0000 7f6bfb6cd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T03:34:37.462 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:34:37.571 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:37.357+0000 7f6bfb6cd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T03:34:37.619 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:37.356+0000 7f1ee7009200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T03:34:37.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:37 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' 2024-04-13T03:34:37.873 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:37.619+0000 7f1ee7009200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T03:34:37.873 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:37.728+0000 7f1ee7009200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T03:34:37.873 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:37.814+0000 7f1ee7009200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T03:34:37.913 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:37 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' 2024-04-13T03:34:37.914 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:37.620+0000 7f6bfb6cd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T03:34:37.914 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:37.730+0000 7f6bfb6cd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T03:34:37.914 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:37.815+0000 7f6bfb6cd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T03:34:37.914 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:37.912+0000 7f6bfb6cd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T03:34:37.978 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:37 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:37.649+0000 7fee4853d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-13T03:34:37.978 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:37 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:37.780+0000 7fee4853d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-13T03:34:37.978 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:37 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' 2024-04-13T03:34:38.232 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:37 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:37.976+0000 7fee4853d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-13T03:34:38.232 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:38.113+0000 7fee4853d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-13T03:34:38.321 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:38 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:38.088+0000 7f6bfb6cd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T03:34:38.372 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:37.911+0000 7f1ee7009200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T03:34:38.373 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:38 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:38.088+0000 7f1ee7009200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T03:34:38.529 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:38.231+0000 7fee4853d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-13T03:34:38.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:38.592+0000 7fee4853d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-13T03:34:38.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:38.741+0000 7fee4853d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-13T03:34:39.238 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:38.865+0000 7fee4853d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-13T03:34:39.239 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:38.998+0000 7fee4853d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-13T03:34:39.321 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:39.040+0000 7f6bfb6cd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T03:34:39.321 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:39.170+0000 7f6bfb6cd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T03:34:39.372 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:39.044+0000 7f1ee7009200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T03:34:39.373 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:39.173+0000 7f1ee7009200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T03:34:39.508 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:39.237+0000 7fee4853d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-13T03:34:39.786 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:39.503+0000 7f6bfb6cd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T03:34:39.786 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:39.592+0000 7f6bfb6cd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T03:34:39.786 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:39.785+0000 7f6bfb6cd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:34:39.790 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:39.507+0000 7f1ee7009200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T03:34:39.790 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:39.595+0000 7f1ee7009200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T03:34:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.882+0000 7f666ac00700 1 -- 172.21.15.16:0/4064173420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 msgr2=0x7f666410ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.882+0000 7f666ac00700 1 --2- 172.21.15.16:0/4064173420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 0x7f666410ef70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6650009990 tx=0x7f6650009ca0 comp rx=0 tx=0).stop 2024-04-13T03:34:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.882+0000 7f666ac00700 1 -- 172.21.15.16:0/4064173420 shutdown_connections 2024-04-13T03:34:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.882+0000 7f666ac00700 1 --2- 172.21.15.16:0/4064173420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f666410f4b0 0x7f66641118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:39.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.882+0000 7f666ac00700 1 --2- 172.21.15.16:0/4064173420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6664074ce0 0x7f66640750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.882+0000 7f666ac00700 1 --2- 172.21.15.16:0/4064173420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 0x7f666410ef70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.882+0000 7f666ac00700 1 -- 172.21.15.16:0/4064173420 >> 172.21.15.16:0/4064173420 conn(0x7f6664079c10 msgr2=0x7f666407a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.882+0000 7f666ac00700 1 -- 172.21.15.16:0/4064173420 shutdown_connections 2024-04-13T03:34:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.882+0000 7f666ac00700 1 -- 172.21.15.16:0/4064173420 wait complete. 2024-04-13T03:34:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.883+0000 7f666ac00700 1 Processor -- start 2024-04-13T03:34:39.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.883+0000 7f666ac00700 1 -- start start 2024-04-13T03:34:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.883+0000 7f666ac00700 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6664074ce0 0x7f666419b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:39.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f666ac00700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 0x7f666419ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f666ac00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f666410f4b0 0x7f666419fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f666ac00700 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6664115030 con 0x7f6664074ce0 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f666ac00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6664114eb0 con 0x7f666410f4b0 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f666ac00700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f66641151b0 con 0x7f6664075600 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6663400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 0x7f666419ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6663400700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 0x7f666419ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:52356/0 (socket says 172.21.15.16:52356) 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6668c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f666410f4b0 0x7f666419fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6663400700 1 -- 172.21.15.16:0/2696218881 learned_addr learned my addr 172.21.15.16:0/2696218881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6663e00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6664074ce0 0x7f666419b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6663400700 1 -- 172.21.15.16:0/2696218881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f666410f4b0 msgr2=0x7f666419fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6663400700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f666410f4b0 0x7f666419fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6663400700 1 -- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6664074ce0 msgr2=0x7f666419b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:39.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6663400700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6664074ce0 0x7f666419b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.884+0000 7f6663400700 1 -- 172.21.15.16:0/2696218881 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66641a0480 con 0x7f6664075600 2024-04-13T03:34:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.885+0000 7f6663400700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 0x7f666419ba00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6650000c00 tx=0x7f665000bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.885+0000 7f6660c00700 1 -- 172.21.15.16:0/2696218881 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6650027070 con 0x7f6664075600 2024-04-13T03:34:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.885+0000 7f6660c00700 1 -- 172.21.15.16:0/2696218881 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66500056d0 con 0x7f6664075600 2024-04-13T03:34:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.885+0000 7f6660c00700 1 -- 172.21.15.16:0/2696218881 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6650022e10 con 0x7f6664075600 2024-04-13T03:34:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.885+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66641a0710 con 0x7f6664075600 2024-04-13T03:34:39.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.885+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66641a7ff0 con 0x7f6664075600 2024-04-13T03:34:39.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.886+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66640697a0 con 0x7f6664075600 2024-04-13T03:34:39.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.889+0000 7f6660c00700 1 -- 172.21.15.16:0/2696218881 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f665002a070 con 0x7f6664075600 2024-04-13T03:34:39.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.890+0000 7f6660c00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f66480a3d10 0x7f66480a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:39.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.890+0000 7f6663e00700 1 -- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f66480a3d10 msgr2=0x7f66480a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1721562410 2024-04-13T03:34:39.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.890+0000 7f6660c00700 1 -- 172.21.15.16:0/2696218881 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f66500d1420 con 0x7f6664075600 2024-04-13T03:34:39.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.890+0000 7f6663e00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f66480a3d10 0x7f66480a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:34:39.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:39.894+0000 7f6660c00700 1 -- 172.21.15.16:0/2696218881 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6650013070 con 0x7f6664075600 2024-04-13T03:34:40.071 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:39.970+0000 7f6bfb6cd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T03:34:40.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.091+0000 7f6663e00700 1 -- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f66480a3d10 msgr2=0x7f66480a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1721562410 2024-04-13T03:34:40.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.091+0000 7f6663e00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f66480a3d10 0x7f66480a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-13T03:34:40.123 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:39.790+0000 7f1ee7009200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:34:40.123 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:39.976+0000 7f1ee7009200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T03:34:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.163+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f666419c7d0 con 0x7f6664075600 2024-04-13T03:34:40.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.163+0000 7f6660c00700 1 -- 172.21.15.16:0/2696218881 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+88 (secure 0 0 0) 0x7f665009dac0 con 0x7f6664075600 2024-04-13T03:34:40.164 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f66480a3d10 msgr2=0x7f66480a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f66480a3d10 0x7f66480a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 msgr2=0x7f666419ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 0x7f666419ba00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6650000c00 tx=0x7f665000bab0 comp rx=0 tx=0).stop 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 shutdown_connections 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f66480a3d10 0x7f66480a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f666410f4b0 0x7f666419fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6664074ce0 0x7f666419b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 --2- 172.21.15.16:0/2696218881 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6664075600 0x7f666419ba00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 >> 172.21.15.16:0/2696218881 conn(0x7f6664079c10 msgr2=0x7f66640ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.166+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 shutdown_connections 2024-04-13T03:34:40.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:40.167+0000 7f666ac00700 1 -- 172.21.15.16:0/2696218881 wait complete. 2024-04-13T03:34:40.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:40 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2696218881' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-13T03:34:40.514 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:40 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2696218881' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-13T03:34:40.515 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:40.257+0000 7f6bfb6cd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:34:40.518 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:40 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:40.257+0000 7f1ee7009200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:34:40.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:40 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2696218881' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-13T03:34:40.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:40.514+0000 7f6bfb6cd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T03:34:40.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:40.749+0000 7f6bfb6cd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T03:34:40.834 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:40 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:40.651+0000 7fee4853d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-13T03:34:40.834 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:40 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:40.833+0000 7fee4853d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-13T03:34:40.872 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:40 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:40.518+0000 7f1ee7009200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T03:34:40.873 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:40 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:40.755+0000 7f1ee7009200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T03:34:41.094 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-13T03:34:41.094 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-13T03:34:41.094 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-04-13T03:34:41.094 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-13T03:34:41.094 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-13T03:34:41.094 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-13T03:34:41.094 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-13T03:34:41.094 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-13T03:34:41.094 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-13T03:34:41.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:41.095 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-04-13T03:34:41.153 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-13T03:34:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-13T03:34:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-13T03:34:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-13T03:34:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-13T03:34:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-13T03:34:41.153 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-13T03:34:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-04-13T03:34:41.153 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 41867sec preferred_lft 41867sec 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-04-13T03:34:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-04-13T03:34:41.156 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-13T03:34:41.156 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:34:41.156 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:34:41.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-04-13T03:34:41.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-04-13T03:34:41.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-04-13T03:34:41.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-13T03:34:41.156 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:34:41.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:34:41.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:41.277+0000 7fee4853d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-13T03:34:41.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:34:41.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:41.418 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-13T03:34:41.486 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-04-13T03:34:41.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:41.490 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-04-13T03:34:41.534 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-04-13T03:34:41.534 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-13T03:34:41.534 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-13T03:34:41.534 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-13T03:34:41.534 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-13T03:34:41.535 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:34:41.535 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:34:41.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-13T03:34:41.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-13T03:34:41.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-13T03:34:41.535 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:34:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:34:41.693 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:41.408+0000 7fee4853d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-13T03:34:41.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:34:41.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:41.821 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-13T03:34:41.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:41 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:41.546+0000 7f6bfb6cd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T03:34:41.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:41.868 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-04-13T03:34:41.872 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:41 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:41.552+0000 7f1ee7009200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T03:34:41.956 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:34:41.956 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:34:41.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-13T03:34:41.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-13T03:34:41.956 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:34:41.960 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:41.692+0000 7fee4853d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:34:41.961 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:41.960+0000 7fee4853d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-13T03:34:42.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:34:42.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:34:42.146 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-13T03:34:42.146 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:34:42.146 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:34:42.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-13T03:34:42.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-13T03:34:42.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-13T03:34:42.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-13T03:34:42.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-13T03:34:42.147 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:34:42.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:34:42.430 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:42.115+0000 7f6bfb6cd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T03:34:42.430 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:42.213+0000 7f6bfb6cd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T03:34:42.430 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y[37233]: 2024-04-13T03:34:42.429+0000 7f6bfb6cd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T03:34:42.438 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:42.127+0000 7f1ee7009200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T03:34:42.439 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:42.225+0000 7f1ee7009200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T03:34:42.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:34:42.447 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-13T03:34:42.448 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-13T03:34:42.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-04-13T03:34:42.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-04-13T03:34:42.448 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-13T03:34:42.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:34:42.582 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:42.373+0000 7fee4853d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: Standby manager daemon y restarted 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: Standby manager daemon y started 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: Standby manager daemon z restarted 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: Standby manager daemon z started 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-13T03:34:42.583 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:34:42.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-13T03:34:42.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-mon[31772]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:34:42.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:34:42.675 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-13T03:34:42.675 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-13T03:34:42.717 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-13T03:34:42.717 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-13T03:34:42.717 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-13T03:34:42.803 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-13T03:34:42.803 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-13T03:34:42.803 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/0eee58cc-391c-4f12-b842-2f4e801dfa8e 2024-04-13T03:34:42.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:42.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/0eee58cc-391c-4f12-b842-2f4e801dfa8e /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-04-13T03:34:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: Standby manager daemon y restarted 2024-04-13T03:34:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: Standby manager daemon y started 2024-04-13T03:34:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-13T03:34:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:34:42.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-13T03:34:42.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:34:42.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: Standby manager daemon z restarted 2024-04-13T03:34:42.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: Standby manager daemon z started 2024-04-13T03:34:42.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-13T03:34:42.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:34:42.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-13T03:34:42.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:42 smithi119 ceph-mon[35835]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:34:42.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:42 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:42.720+0000 7fee4853d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-13T03:34:42.873 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z[38262]: 2024-04-13T03:34:42.438+0000 7f1ee7009200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T03:34:42.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: Standby manager daemon y restarted 2024-04-13T03:34:42.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: Standby manager daemon y started 2024-04-13T03:34:42.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-13T03:34:42.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:34:42.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-13T03:34:42.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.119:0/3549372633' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:34:42.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: Standby manager daemon z restarted 2024-04-13T03:34:42.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: Standby manager daemon z started 2024-04-13T03:34:42.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-13T03:34:42.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-13T03:34:42.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-13T03:34:42.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:42 smithi188 ceph-mon[36398]: from='mgr.? 172.21.15.188:0/159592880' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-13T03:34:42.978 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-04-13T03:34:42.978 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-04-13T03:34:42.978 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2024-04-13T03:34:43.256 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:43 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:43.048+0000 7fee4853d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-13T03:34:43.764 INFO:teuthology.orchestra.run.smithi016.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-04-13T03:34:43.764 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-04-13T03:34:43.764 INFO:teuthology.orchestra.run.smithi016.stdout:invalid new device string format 2024-04-13T03:34:43.764 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: resolved to: "172.21.15.16:6789,172.21.15.119:6789,172.21.15.188:6789" 2024-04-13T03:34:43.764 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.16:6789,172.21.15.119:6789,172.21.15.188:6789:/volumes/qa/sv_1/0eee58cc-391c-4f12-b842-2f4e801dfa8e 2024-04-13T03:34:43.765 INFO:teuthology.orchestra.run.smithi016.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=e6bf0314-f944-11ee-bc8d-c7b262605968" will pass to kernel 2024-04-13T03:34:43.771 INFO:teuthology.orchestra.run.smithi016.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-13T03:34:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: You just mounted a file system that supports labels which does not 2024-04-13T03:34:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-13T03:34:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: applications will generate AVC messages and not be allowed access to 2024-04-13T03:34:43.771 INFO:teuthology.orchestra.run.smithi016.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-13T03:34:43.771 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:43 smithi016 ceph-mon[31772]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-04-13T03:34:43.773 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-13T03:34:43.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:34:43.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-13T03:34:43.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:43 smithi119 ceph-mon[35835]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-04-13T03:34:43.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:43 smithi188 ceph-mon[36398]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-04-13T03:34:43.902 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-13T03:34:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-13T03:34:44.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:34:44.020 INFO:teuthology.orchestra.run.smithi016.stdout:client25471 2024-04-13T03:34:44.021 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:34:44.021 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-13T03:34:44.064 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-13T03:34:44.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-13T03:34:44.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:34:44.227 INFO:teuthology.orchestra.run.smithi016.stdout:client25471 2024-04-13T03:34:44.227 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-13T03:34:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-13T03:34:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:34:44.349 INFO:teuthology.orchestra.run.smithi016.stdout:client25471 2024-04-13T03:34:44.350 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968.client25471/status') 2024-04-13T03:34:44.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-13T03:34:44.487 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:44 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:44.162+0000 7fee4853d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-13T03:34:44.495 INFO:teuthology.orchestra.run.smithi016.stdout:instance: client.25471 (3)172.21.15.16:0/2026305636 2024-04-13T03:34:44.495 INFO:teuthology.orchestra.run.smithi016.stdout:blocklisted: false 2024-04-13T03:34:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records in 2024-04-13T03:34:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:0+1 records out 2024-04-13T03:34:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:71 bytes copied, 6.6648e-05 s, 1.1 MB/s 2024-04-13T03:34:44.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-13T03:34:44.508 INFO:teuthology.run_tasks:Running task exec... 2024-04-13T03:34:44.517 INFO:teuthology.task.exec:Executing custom commands... 2024-04-13T03:34:44.517 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T03:34:44.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-13T03:34:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.719+0000 7f6c14400640 1 -- 172.21.15.16:0/2705421782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 msgr2=0x7f6c0c10c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.719+0000 7f6c14400640 1 --2- 172.21.15.16:0/2705421782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 0x7f6c0c10c690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009be0 tx=0x7f6c0402bef0 comp rx=0 tx=0).stop 2024-04-13T03:34:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.719+0000 7f6c14400640 1 -- 172.21.15.16:0/2705421782 shutdown_connections 2024-04-13T03:34:44.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.719+0000 7f6c14400640 1 --2- 172.21.15.16:0/2705421782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 0x7f6c0c10c690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:44.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.719+0000 7f6c14400640 1 --2- 172.21.15.16:0/2705421782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c105440 0x7f6c0c1058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:44.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.719+0000 7f6c14400640 1 --2- 172.21.15.16:0/2705421782 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c104250 0x7f6c0c104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:44.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.719+0000 7f6c14400640 1 -- 172.21.15.16:0/2705421782 >> 172.21.15.16:0/2705421782 conn(0x7f6c0c0ff7e0 msgr2=0x7f6c0c101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:44.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.719+0000 7f6c14400640 1 -- 172.21.15.16:0/2705421782 shutdown_connections 2024-04-13T03:34:44.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.719+0000 7f6c14400640 1 -- 172.21.15.16:0/2705421782 wait complete. 2024-04-13T03:34:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.720+0000 7f6c14400640 1 Processor -- start 2024-04-13T03:34:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.720+0000 7f6c14400640 1 -- start start 2024-04-13T03:34:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c14400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c104250 0x7f6c0c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c14400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105440 0x7f6c0c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c14400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 0x7f6c0c1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:44.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c14400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c10ff30 con 0x7f6c0c104250 2024-04-13T03:34:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c14400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c10fdb0 con 0x7f6c0c105e00 2024-04-13T03:34:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c14400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1100b0 con 0x7f6c0c105440 2024-04-13T03:34:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c12400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 0x7f6c0c1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c11000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105440 0x7f6c0c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c12400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 0x7f6c0c1a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:46082/0 (socket says 172.21.15.16:46082) 2024-04-13T03:34:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c12400640 1 -- 172.21.15.16:0/4235554361 learned_addr learned my addr 172.21.15.16:0/4235554361 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:44.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.721+0000 7f6c11a00640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c104250 0x7f6c0c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c11000640 1 -- 172.21.15.16:0/4235554361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 msgr2=0x7f6c0c1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c11000640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 0x7f6c0c1a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c11000640 1 -- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c104250 msgr2=0x7f6c0c19a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c11000640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c104250 0x7f6c0c19a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c11000640 1 -- 172.21.15.16:0/4235554361 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c1a2020 con 0x7f6c0c105440 2024-04-13T03:34:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c12400640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 0x7f6c0c1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c11000640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105440 0x7f6c0c19a940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800e790 tx=0x7f6bf800ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c0a000640 1 -- 172.21.15.16:0/4235554361 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf800eea0 con 0x7f6c0c105440 2024-04-13T03:34:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c1a2310 con 0x7f6c0c105440 2024-04-13T03:34:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c0a000640 1 -- 172.21.15.16:0/4235554361 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bf800ccc0 con 0x7f6c0c105440 2024-04-13T03:34:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c0a000640 1 -- 172.21.15.16:0/4235554361 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8018760 con 0x7f6c0c105440 2024-04-13T03:34:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.722+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c1a28b0 con 0x7f6c0c105440 2024-04-13T03:34:44.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.724+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c10b4a0 con 0x7f6c0c105440 2024-04-13T03:34:44.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.726+0000 7f6c0a000640 1 -- 172.21.15.16:0/4235554361 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7f6bf8010430 con 0x7f6c0c105440 2024-04-13T03:34:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.727+0000 7f6c0a000640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6bf40a3ca0 0x7f6bf40a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:44.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.727+0000 7f6c11a00640 1 -- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6bf40a3ca0 msgr2=0x7f6bf40a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1721562410 2024-04-13T03:34:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.727+0000 7f6c0a000640 1 -- 172.21.15.16:0/4235554361 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6bf80d0970 con 0x7f6c0c105440 2024-04-13T03:34:44.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.728+0000 7f6c11a00640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6bf40a3ca0 0x7f6bf40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:34:44.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.731+0000 7f6c0a000640 1 -- 172.21.15.16:0/4235554361 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bf809c930 con 0x7f6c0c105440 2024-04-13T03:34:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.929+0000 7f6c11a00640 1 -- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6bf40a3ca0 msgr2=0x7f6bf40a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1721562410 2024-04-13T03:34:44.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.929+0000 7f6c11a00640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6bf40a3ca0 0x7f6bf40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-13T03:34:44.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:44.981+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f6c0c062bc0 con 0x7f6c0c105440 2024-04-13T03:34:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.039+0000 7f6c0a000640 1 -- 172.21.15.16:0/4235554361 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v27) ==== 117+0+0 (secure 0 0 0) 0x7f6bf80a0c30 con 0x7f6c0c105440 2024-04-13T03:34:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6bf40a3ca0 msgr2=0x7f6bf40a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:34:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6bf40a3ca0 0x7f6bf40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105440 msgr2=0x7f6c0c19a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105440 0x7f6c0c19a940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800e790 tx=0x7f6bf800ec50 comp rx=0 tx=0).stop 2024-04-13T03:34:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 shutdown_connections 2024-04-13T03:34:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f6bf40a3ca0 0x7f6bf40a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c0c105e00 0x7f6c0c1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c0c105440 0x7f6c0c19a940 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 --2- 172.21.15.16:0/4235554361 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c104250 0x7f6c0c19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 >> 172.21.15.16:0/4235554361 conn(0x7f6c0c0ff7e0 msgr2=0x7f6c0c108670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:45.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 shutdown_connections 2024-04-13T03:34:45.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.043+0000 7f6c14400640 1 -- 172.21.15.16:0/4235554361 wait complete. 2024-04-13T03:34:45.063 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-13T03:34:45.077 INFO:teuthology.run_tasks:Running task exec... 2024-04-13T03:34:45.088 INFO:teuthology.task.exec:Executing custom commands... 2024-04-13T03:34:45.088 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T03:34:45.089 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-13T03:34:45.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4235554361' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-13T03:34:45.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-13T03:34:45.166 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:44 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:44.908+0000 7fee4853d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-13T03:34:45.166 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.053+0000 7fee4853d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-13T03:34:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.292+0000 7f23c8a00640 1 -- 172.21.15.16:0/2169060838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23c4101e90 msgr2=0x7f23c41022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.292+0000 7f23c8a00640 1 --2- 172.21.15.16:0/2169060838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23c4101e90 0x7f23c41022b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f23b4009860 tx=0x7f23b402bb90 comp rx=0 tx=0).stop 2024-04-13T03:34:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.293+0000 7f23c8a00640 1 -- 172.21.15.16:0/2169060838 shutdown_connections 2024-04-13T03:34:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.293+0000 7f23c8a00640 1 --2- 172.21.15.16:0/2169060838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c4103a30 0x7f23c410a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.293+0000 7f23c8a00640 1 --2- 172.21.15.16:0/2169060838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23c4103070 0x7f23c41034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.293+0000 7f23c8a00640 1 --2- 172.21.15.16:0/2169060838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23c4101e90 0x7f23c41022b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.293+0000 7f23c8a00640 1 -- 172.21.15.16:0/2169060838 >> 172.21.15.16:0/2169060838 conn(0x7f23c40fd450 msgr2=0x7f23c40ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.293+0000 7f23c8a00640 1 -- 172.21.15.16:0/2169060838 shutdown_connections 2024-04-13T03:34:45.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.293+0000 7f23c8a00640 1 -- 172.21.15.16:0/2169060838 wait complete. 2024-04-13T03:34:45.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.294+0000 7f23c8a00640 1 Processor -- start 2024-04-13T03:34:45.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.294+0000 7f23c8a00640 1 -- start start 2024-04-13T03:34:45.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c8a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c4101e90 0x7f23c419a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c8a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23c4103070 0x7f23c419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c8a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23c4103a30 0x7f23c41a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c8a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23c410db00 con 0x7f23c4101e90 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c8a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f23c410d980 con 0x7f23c4103a30 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c8a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f23c410dc80 con 0x7f23c4103070 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c4101e90 0x7f23c419a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c2800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c4101e90 0x7f23c419a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45158/0 (socket says 172.21.15.16:45158) 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c2800640 1 -- 172.21.15.16:0/194997548 learned_addr learned my addr 172.21.15.16:0/194997548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c3200640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23c4103a30 0x7f23c41a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.295+0000 7f23c1e00640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23c4103070 0x7f23c419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23c2800640 1 -- 172.21.15.16:0/194997548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23c4103070 msgr2=0x7f23c419a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23c2800640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23c4103070 0x7f23c419a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23c2800640 1 -- 172.21.15.16:0/194997548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23c4103a30 msgr2=0x7f23c41a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:45.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23c2800640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23c4103a30 0x7f23c41a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23c2800640 1 -- 172.21.15.16:0/194997548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c41a1fb0 con 0x7f23c4101e90 2024-04-13T03:34:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23c3200640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23c4103a30 0x7f23c41a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:34:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23c2800640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c4101e90 0x7f23c419a390 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f23b40349e0 tx=0x7f23b4032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b4041020 con 0x7f23c4101e90 2024-04-13T03:34:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23c8a00640 1 -- 172.21.15.16:0/194997548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23c41a2240 con 0x7f23c4101e90 2024-04-13T03:34:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23b402ca10 con 0x7f23c4101e90 2024-04-13T03:34:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.296+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b4039040 con 0x7f23c4101e90 2024-04-13T03:34:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.297+0000 7f23c8a00640 1 -- 172.21.15.16:0/194997548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23c41a2780 con 0x7f23c4101e90 2024-04-13T03:34:45.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.298+0000 7f23c8a00640 1 -- 172.21.15.16:0/194997548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23c41a29e0 con 0x7f23c4101e90 2024-04-13T03:34:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.300+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7f23b4044020 con 0x7f23c4101e90 2024-04-13T03:34:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.302+0000 7f23bb400640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f23a80a3bf0 0x7f23a80a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.302+0000 7f23c1e00640 1 -- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f23a80a3bf0 msgr2=0x7f23a80a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1721562410 2024-04-13T03:34:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.302+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f23b40f2040 con 0x7f23c4101e90 2024-04-13T03:34:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.302+0000 7f23c1e00640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f23a80a3bf0 0x7f23a80a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-13T03:34:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.307+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23b404b020 con 0x7f23c4101e90 2024-04-13T03:34:45.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:45 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4235554361' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-13T03:34:45.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:45 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-13T03:34:45.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:45 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4235554361' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-13T03:34:45.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:45 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-13T03:34:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.390+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f23b40042c0 con 0x7f23c4101e90 2024-04-13T03:34:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.391+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f23a80a3bf0 msgr2=0x7f23a80a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-13T03:34:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:45.391+0000 7f23bb400640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1721562410,v1:172.21.15.16:6801/1721562410] conn(0x7f23a80a3bf0 0x7f23a80a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:45.586 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.360+0000 7fee4853d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-13T03:34:45.855 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.696+0000 7fed9ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:45.855 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.696+0000 7fed9ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:45.855 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.696+0000 7fed9ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:45.855 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.696+0000 7fed9ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:45.855 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.696+0000 7fed9ae00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:45.855 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.697+0000 7fed9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:45.856 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.697+0000 7fed9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:45.856 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.697+0000 7fed9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:45.856 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.697+0000 7fed9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:45.856 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:34:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:34:45.697+0000 7fed9e000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:34:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: Active manager daemon x restarted 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: Activating manager daemon x 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-13T03:34:46.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: osdmap e95: 12 total, 12 up, 12 in 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: mgrmap e21: x(active, starting, since 0.029127s), standbys: y, z 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-13T03:34:46.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-13T03:34:46.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: Manager daemon x is now available 2024-04-13T03:34:46.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:46.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-13T03:34:46.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-13T03:34:46.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-13T03:34:46.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:46 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-13T03:34:46.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-13T03:34:46.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: Active manager daemon x restarted 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: Activating manager daemon x 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-13T03:34:46.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: osdmap e95: 12 total, 12 up, 12 in 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: mgrmap e21: x(active, starting, since 0.029127s), standbys: y, z 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:34:46.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: Manager daemon x is now available 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-13T03:34:46.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-13T03:34:46.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-13T03:34:46.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:46 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-13T03:34:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-13T03:34:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:34:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: Active manager daemon x restarted 2024-04-13T03:34:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: Activating manager daemon x 2024-04-13T03:34:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T03:34:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T03:34:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T03:34:46.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: osdmap e95: 12 total, 12 up, 12 in 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: mgrmap e21: x(active, starting, since 0.029127s), standbys: y, z 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-13T03:34:46.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-13T03:34:46.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: Manager daemon x is now available 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-13T03:34:46.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:46 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-13T03:34:46.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.409+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f23b40ba5c0 con 0x7f23c4101e90 2024-04-13T03:34:46.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.410+0000 7f23bb400640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f23a8052120 0x7f23a8052500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:46.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.410+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 --> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f23b40f4290 con 0x7f23a8052120 2024-04-13T03:34:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.416+0000 7f23c1e00640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f23a8052120 0x7f23a8052500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:46.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.425+0000 7f23c1e00640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f23a8052120 0x7f23a8052500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f23c419b8a0 tx=0x7f23ac005f90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.537+0000 7f23bb400640 1 -- 172.21.15.16:0/194997548 <== mgr.25450 v2:172.21.15.16:6800/1691605870 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f23b40f4290 con 0x7f23a8052120 2024-04-13T03:34:46.539 INFO:teuthology.orchestra.run.smithi016.stdout:Schedule set for path / 2024-04-13T03:34:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.540+0000 7f23c8a00640 1 -- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f23a8052120 msgr2=0x7f23a8052500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.540+0000 7f23c8a00640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f23a8052120 0x7f23a8052500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f23c419b8a0 tx=0x7f23ac005f90 comp rx=0 tx=0).stop 2024-04-13T03:34:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.540+0000 7f23c8a00640 1 -- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c4101e90 msgr2=0x7f23c419a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.540+0000 7f23c8a00640 1 --2- 172.21.15.16:0/194997548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23c4101e90 0x7f23c419a390 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f23b40349e0 tx=0x7f23b4032430 comp rx=0 tx=0).stop 2024-04-13T03:34:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.540+0000 7f23c3200640 1 -- 172.21.15.16:0/194997548 reap_dead start 2024-04-13T03:34:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.540+0000 7f23c8a00640 1 -- 172.21.15.16:0/194997548 shutdown_connections 2024-04-13T03:34:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.540+0000 7f23c8a00640 1 -- 172.21.15.16:0/194997548 >> 172.21.15.16:0/194997548 conn(0x7f23c40fd450 msgr2=0x7f23c40741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.541+0000 7f23c8a00640 1 -- 172.21.15.16:0/194997548 shutdown_connections 2024-04-13T03:34:46.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.541+0000 7f23c8a00640 1 -- 172.21.15.16:0/194997548 wait complete. 2024-04-13T03:34:46.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-13T03:34:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.775+0000 7f041da00640 1 -- 172.21.15.16:0/3379511230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 msgr2=0x7f0418104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.775+0000 7f041da00640 1 --2- 172.21.15.16:0/3379511230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 0x7f0418104660 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0408009860 tx=0x7f040802bb70 comp rx=0 tx=0).stop 2024-04-13T03:34:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.775+0000 7f041da00640 1 -- 172.21.15.16:0/3379511230 shutdown_connections 2024-04-13T03:34:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.775+0000 7f041da00640 1 --2- 172.21.15.16:0/3379511230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0418105df0 0x7f041810c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.775+0000 7f041da00640 1 --2- 172.21.15.16:0/3379511230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0418105430 0x7f04181058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.775+0000 7f041da00640 1 --2- 172.21.15.16:0/3379511230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 0x7f0418104660 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.775+0000 7f041da00640 1 -- 172.21.15.16:0/3379511230 >> 172.21.15.16:0/3379511230 conn(0x7f04180ff7b0 msgr2=0x7f0418101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.776+0000 7f041da00640 1 -- 172.21.15.16:0/3379511230 shutdown_connections 2024-04-13T03:34:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.776+0000 7f041da00640 1 -- 172.21.15.16:0/3379511230 wait complete. 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f041da00640 1 Processor -- start 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f041da00640 1 -- start start 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f041da00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 0x7f041819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f041da00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0418105430 0x7f041819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f041da00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0418105df0 0x7f04181a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f041da00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f041810fed0 con 0x7f0418105430 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f041da00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f041810fd50 con 0x7f0418104240 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f041da00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f0418110050 con 0x7f0418105df0 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f0416a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 0x7f041819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f040fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0418105430 0x7f041819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f0416a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 0x7f041819a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:46242/0 (socket says 172.21.15.16:46242) 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f0416a00640 1 -- 172.21.15.16:0/3208822675 learned_addr learned my addr 172.21.15.16:0/3208822675 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.777+0000 7f0417400640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0418105df0 0x7f04181a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f0416a00640 1 -- 172.21.15.16:0/3208822675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0418105df0 msgr2=0x7f04181a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f0416a00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0418105df0 0x7f04181a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f0416a00640 1 -- 172.21.15.16:0/3208822675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0418105430 msgr2=0x7f041819aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f0416a00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0418105430 0x7f041819aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f0416a00640 1 -- 172.21.15.16:0/3208822675 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04181a20e0 con 0x7f0418104240 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f0417400640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0418105df0 0x7f04181a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f0416a00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 0x7f041819a4c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f04080349e0 tx=0x7f04080042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f040f400640 1 -- 172.21.15.16:0/3208822675 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0408041020 con 0x7f0418104240 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f040f400640 1 -- 172.21.15.16:0/3208822675 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04080323b0 con 0x7f0418104240 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04181a2370 con 0x7f0418104240 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.778+0000 7f040f400640 1 -- 172.21.15.16:0/3208822675 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0408039040 con 0x7f0418104240 2024-04-13T03:34:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.779+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04181a28b0 con 0x7f0418104240 2024-04-13T03:34:46.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.780+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f041810b500 con 0x7f0418104240 2024-04-13T03:34:46.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.783+0000 7f040f400640 1 -- 172.21.15.16:0/3208822675 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f0408044020 con 0x7f0418104240 2024-04-13T03:34:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.784+0000 7f040f400640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f03fc0a3ae0 0x7f03fc0a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.785+0000 7f040f400640 1 -- 172.21.15.16:0/3208822675 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f04080f1df0 con 0x7f0418104240 2024-04-13T03:34:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.785+0000 7f040fe00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f03fc0a3ae0 0x7f03fc0a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.786+0000 7f040fe00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f03fc0a3ae0 0x7f03fc0a5fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f041819b9d0 tx=0x7f0400009260 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:46.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:46.793+0000 7f040f400640 1 -- 172.21.15.16:0/3208822675 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f040804b020 con 0x7f0418104240 2024-04-13T03:34:47.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.048+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 --> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f0418062560 con 0x7f03fc0a3ae0 2024-04-13T03:34:47.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.180+0000 7f040f400640 1 -- 172.21.15.16:0/3208822675 <== mgr.25450 v2:172.21.15.16:6800/1691605870 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f0418062560 con 0x7f03fc0a3ae0 2024-04-13T03:34:47.182 INFO:teuthology.orchestra.run.smithi016.stdout:Retention added to path / 2024-04-13T03:34:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.183+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f03fc0a3ae0 msgr2=0x7f03fc0a5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.183+0000 7f041da00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f03fc0a3ae0 0x7f03fc0a5fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f041819b9d0 tx=0x7f0400009260 comp rx=0 tx=0).stop 2024-04-13T03:34:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.183+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 msgr2=0x7f041819a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.183+0000 7f041da00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 0x7f041819a4c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f04080349e0 tx=0x7f04080042f0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.184+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 shutdown_connections 2024-04-13T03:34:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.184+0000 7f041da00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f03fc0a3ae0 0x7f03fc0a5fa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.184+0000 7f041da00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0418105df0 0x7f04181a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.184+0000 7f041da00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0418105430 0x7f041819aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.184+0000 7f041da00640 1 --2- 172.21.15.16:0/3208822675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0418104240 0x7f041819a4c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.184+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 >> 172.21.15.16:0/3208822675 conn(0x7f04180ff7b0 msgr2=0x7f04181086d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.184+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 shutdown_connections 2024-04-13T03:34:47.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.184+0000 7f041da00640 1 -- 172.21.15.16:0/3208822675 wait complete. 2024-04-13T03:34:47.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-13T03:34:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.418+0000 7f8963000640 1 -- 172.21.15.16:0/1703248694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c076000 msgr2=0x7f895c074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.418+0000 7f8963000640 1 --2- 172.21.15.16:0/1703248694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c076000 0x7f895c074450 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f894c009900 tx=0x7f894c02bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.419+0000 7f8963000640 1 -- 172.21.15.16:0/1703248694 shutdown_connections 2024-04-13T03:34:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.419+0000 7f8963000640 1 --2- 172.21.15.16:0/1703248694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f895c109ef0 0x7f895c10c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.419+0000 7f8963000640 1 --2- 172.21.15.16:0/1703248694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f895c074a20 0x7f895c074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.419+0000 7f8963000640 1 --2- 172.21.15.16:0/1703248694 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c076000 0x7f895c074450 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.419+0000 7f8963000640 1 -- 172.21.15.16:0/1703248694 >> 172.21.15.16:0/1703248694 conn(0x7f895c0fd580 msgr2=0x7f895c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.419+0000 7f8963000640 1 -- 172.21.15.16:0/1703248694 shutdown_connections 2024-04-13T03:34:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.419+0000 7f8963000640 1 -- 172.21.15.16:0/1703248694 wait complete. 2024-04-13T03:34:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.420+0000 7f8963000640 1 Processor -- start 2024-04-13T03:34:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.420+0000 7f8963000640 1 -- start start 2024-04-13T03:34:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.420+0000 7f8963000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f895c074a20 0x7f895c19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.420+0000 7f8963000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f895c076000 0x7f895c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.420+0000 7f8963000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c109ef0 0x7f895c1a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.420+0000 7f8963000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f895c10feb0 con 0x7f895c109ef0 2024-04-13T03:34:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.420+0000 7f8963000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f895c10fd30 con 0x7f895c076000 2024-04-13T03:34:47.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.420+0000 7f8963000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f895c110030 con 0x7f895c074a20 2024-04-13T03:34:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f895b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f895c076000 0x7f895c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f8961000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c109ef0 0x7f895c1a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f895b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f895c076000 0x7f895c19a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:46260/0 (socket says 172.21.15.16:46260) 2024-04-13T03:34:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f895b400640 1 -- 172.21.15.16:0/2974845467 learned_addr learned my addr 172.21.15.16:0/2974845467 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f895be00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f895c074a20 0x7f895c19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:47.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f895b400640 1 -- 172.21.15.16:0/2974845467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f895c074a20 msgr2=0x7f895c19a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f895b400640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f895c074a20 0x7f895c19a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f895b400640 1 -- 172.21.15.16:0/2974845467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c109ef0 msgr2=0x7f895c1a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f895b400640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c109ef0 0x7f895c1a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.421+0000 7f895b400640 1 -- 172.21.15.16:0/2974845467 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c1a1ff0 con 0x7f895c076000 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.422+0000 7f895b400640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f895c076000 0x7f895c19a910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f894800c920 tx=0x7f894800cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.422+0000 7f8958c00640 1 -- 172.21.15.16:0/2974845467 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948007c70 con 0x7f895c076000 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.422+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895c1a22e0 con 0x7f895c076000 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.422+0000 7f8958c00640 1 -- 172.21.15.16:0/2974845467 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8948007e10 con 0x7f895c076000 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.422+0000 7f8958c00640 1 -- 172.21.15.16:0/2974845467 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8948018890 con 0x7f895c076000 2024-04-13T03:34:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.422+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895c1a2880 con 0x7f895c076000 2024-04-13T03:34:47.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.424+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895c108c20 con 0x7f895c076000 2024-04-13T03:34:47.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.427+0000 7f8958c00640 1 -- 172.21.15.16:0/2974845467 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f8948018a70 con 0x7f895c076000 2024-04-13T03:34:47.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.429+0000 7f8958c00640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89400a3e40 0x7f89400a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:47.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.429+0000 7f8958c00640 1 -- 172.21.15.16:0/2974845467 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7f89480d0bd0 con 0x7f895c076000 2024-04-13T03:34:47.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.429+0000 7f895be00640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89400a3e40 0x7f89400a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:47.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.430+0000 7f895be00640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89400a3e40 0x7f89400a6300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f894c02c7b0 tx=0x7f894c005fd0 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:47.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.433+0000 7f8958c00640 1 -- 172.21.15.16:0/2974845467 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8948014030 con 0x7f895c076000 2024-04-13T03:34:47.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:47 smithi016 ceph-mon[31772]: [13/Apr/2024:03:34:45] ENGINE Bus STARTING 2024-04-13T03:34:47.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:47 smithi016 ceph-mon[31772]: [13/Apr/2024:03:34:46] ENGINE Serving on https://172.21.15.16:7150 2024-04-13T03:34:47.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:47 smithi016 ceph-mon[31772]: [13/Apr/2024:03:34:46] ENGINE Serving on http://172.21.15.16:8765 2024-04-13T03:34:47.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:47 smithi016 ceph-mon[31772]: [13/Apr/2024:03:34:46] ENGINE Bus STARTED 2024-04-13T03:34:47.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:47 smithi016 ceph-mon[31772]: mgrmap e22: x(active, since 1.04326s), standbys: y, z 2024-04-13T03:34:47.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:47 smithi016 ceph-mon[31772]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:47.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:47 smithi016 ceph-mon[31772]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.684+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 --> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f895c062400 con 0x7f89400a3e40 2024-04-13T03:34:47.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.708+0000 7f8958c00640 1 -- 172.21.15.16:0/2974845467 <== mgr.25450 v2:172.21.15.16:6800/1691605870 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f895c062400 con 0x7f89400a3e40 2024-04-13T03:34:47.710 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-13T00:00:00", "created": "2024-04-13T03:34:46", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-13T03:34:47.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89400a3e40 msgr2=0x7f89400a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89400a3e40 0x7f89400a6300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f894c02c7b0 tx=0x7f894c005fd0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f895c076000 msgr2=0x7f895c19a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f895c076000 0x7f895c19a910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f894800c920 tx=0x7f894800cde0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 shutdown_connections 2024-04-13T03:34:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89400a3e40 0x7f89400a6300 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f895c109ef0 0x7f895c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f895c076000 0x7f895c19a910 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 --2- 172.21.15.16:0/2974845467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f895c074a20 0x7f895c19a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 >> 172.21.15.16:0/2974845467 conn(0x7f895c0fd580 msgr2=0x7f895c105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 shutdown_connections 2024-04-13T03:34:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.711+0000 7f8963000640 1 -- 172.21.15.16:0/2974845467 wait complete. 2024-04-13T03:34:47.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-13T03:34:47.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:47 smithi119 ceph-mon[35835]: [13/Apr/2024:03:34:45] ENGINE Bus STARTING 2024-04-13T03:34:47.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:47 smithi119 ceph-mon[35835]: [13/Apr/2024:03:34:46] ENGINE Serving on https://172.21.15.16:7150 2024-04-13T03:34:47.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:47 smithi119 ceph-mon[35835]: [13/Apr/2024:03:34:46] ENGINE Serving on http://172.21.15.16:8765 2024-04-13T03:34:47.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:47 smithi119 ceph-mon[35835]: [13/Apr/2024:03:34:46] ENGINE Bus STARTED 2024-04-13T03:34:47.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:47 smithi119 ceph-mon[35835]: mgrmap e22: x(active, since 1.04326s), standbys: y, z 2024-04-13T03:34:47.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:47 smithi119 ceph-mon[35835]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:47.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:47 smithi119 ceph-mon[35835]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:47.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:47 smithi188 ceph-mon[36398]: [13/Apr/2024:03:34:45] ENGINE Bus STARTING 2024-04-13T03:34:47.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:47 smithi188 ceph-mon[36398]: [13/Apr/2024:03:34:46] ENGINE Serving on https://172.21.15.16:7150 2024-04-13T03:34:47.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:47 smithi188 ceph-mon[36398]: [13/Apr/2024:03:34:46] ENGINE Serving on http://172.21.15.16:8765 2024-04-13T03:34:47.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:47 smithi188 ceph-mon[36398]: [13/Apr/2024:03:34:46] ENGINE Bus STARTED 2024-04-13T03:34:47.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:47 smithi188 ceph-mon[36398]: mgrmap e22: x(active, since 1.04326s), standbys: y, z 2024-04-13T03:34:47.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:47 smithi188 ceph-mon[36398]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:47.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:47 smithi188 ceph-mon[36398]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.948+0000 7fd815e00640 1 -- 172.21.15.16:0/4124629463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8101028c0 msgr2=0x7fd810104cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.948+0000 7fd815e00640 1 --2- 172.21.15.16:0/4124629463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8101028c0 0x7fd810104cb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd804009900 tx=0x7fd80402bc30 comp rx=0 tx=0).stop 2024-04-13T03:34:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.949+0000 7fd815e00640 1 -- 172.21.15.16:0/4124629463 shutdown_connections 2024-04-13T03:34:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.949+0000 7fd815e00640 1 --2- 172.21.15.16:0/4124629463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd810107b20 0x7fd810109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.949+0000 7fd815e00640 1 --2- 172.21.15.16:0/4124629463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8101051f0 0x7fd8101075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.949+0000 7fd815e00640 1 --2- 172.21.15.16:0/4124629463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8101028c0 0x7fd810104cb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.949+0000 7fd815e00640 1 -- 172.21.15.16:0/4124629463 >> 172.21.15.16:0/4124629463 conn(0x7fd8100fc2b0 msgr2=0x7fd8100fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.949+0000 7fd815e00640 1 -- 172.21.15.16:0/4124629463 shutdown_connections 2024-04-13T03:34:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.949+0000 7fd815e00640 1 -- 172.21.15.16:0/4124629463 wait complete. 2024-04-13T03:34:47.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.951+0000 7fd815e00640 1 Processor -- start 2024-04-13T03:34:47.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.951+0000 7fd815e00640 1 -- start start 2024-04-13T03:34:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.951+0000 7fd815e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8101028c0 0x7fd810198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd815e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8101051f0 0x7fd8101986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd815e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd810107b20 0x7fd81019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd815e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd81010db10 con 0x7fd8101051f0 2024-04-13T03:34:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd815e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd81010d990 con 0x7fd8101028c0 2024-04-13T03:34:47.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd815e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd81010dc90 con 0x7fd810107b20 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd814a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8101028c0 0x7fd810198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd814a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8101028c0 0x7fd810198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:46292/0 (socket says 172.21.15.16:46292) 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd814a00640 1 -- 172.21.15.16:0/2797785928 learned_addr learned my addr 172.21.15.16:0/2797785928 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd815400640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd810107b20 0x7fd81019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd80fe00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8101051f0 0x7fd8101986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd814a00640 1 -- 172.21.15.16:0/2797785928 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd810107b20 msgr2=0x7fd81019f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd814a00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd810107b20 0x7fd81019f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd814a00640 1 -- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8101051f0 msgr2=0x7fd8101986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd814a00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8101051f0 0x7fd8101986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.952+0000 7fd814a00640 1 -- 172.21.15.16:0/2797785928 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81019fdb0 con 0x7fd8101028c0 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.953+0000 7fd814a00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8101028c0 0x7fd810198190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd804000c00 tx=0x7fd80402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.953+0000 7fd80d600640 1 -- 172.21.15.16:0/2797785928 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804043020 con 0x7fd8101028c0 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.953+0000 7fd80d600640 1 -- 172.21.15.16:0/2797785928 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8040054e0 con 0x7fd8101028c0 2024-04-13T03:34:47.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.953+0000 7fd80d600640 1 -- 172.21.15.16:0/2797785928 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804003bd0 con 0x7fd8101028c0 2024-04-13T03:34:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.953+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8101a0040 con 0x7fd8101028c0 2024-04-13T03:34:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.953+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8101a0540 con 0x7fd8101028c0 2024-04-13T03:34:47.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.955+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8100fdda0 con 0x7fd8101028c0 2024-04-13T03:34:47.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.957+0000 7fd80d600640 1 -- 172.21.15.16:0/2797785928 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fd804046020 con 0x7fd8101028c0 2024-04-13T03:34:47.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.959+0000 7fd80d600640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd7f80a3d10 0x7fd7f80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:34:47.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.959+0000 7fd80d600640 1 -- 172.21.15.16:0/2797785928 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9466+0+0 (secure 0 0 0) 0x7fd8040f2bc0 con 0x7fd8101028c0 2024-04-13T03:34:47.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.959+0000 7fd80fe00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd7f80a3d10 0x7fd7f80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:34:47.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.960+0000 7fd80fe00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd7f80a3d10 0x7fd7f80a61d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd8101996a0 tx=0x7fd7fc005f90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:34:47.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:47.962+0000 7fd80d600640 1 -- 172.21.15.16:0/2797785928 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd80404d020 con 0x7fd8101028c0 2024-04-13T03:34:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.230+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 --> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fd810062400 con 0x7fd7f80a3d10 2024-04-13T03:34:48.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.248+0000 7fd80d600640 1 -- 172.21.15.16:0/2797785928 <== mgr.25450 v2:172.21.15.16:6800/1691605870 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fd810062400 con 0x7fd7f80a3d10 2024-04-13T03:34:48.250 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-04-13T03:34:48.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd7f80a3d10 msgr2=0x7fd7f80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd7f80a3d10 0x7fd7f80a61d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd8101996a0 tx=0x7fd7fc005f90 comp rx=0 tx=0).stop 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8101028c0 msgr2=0x7fd810198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8101028c0 0x7fd810198190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd804000c00 tx=0x7fd80402c650 comp rx=0 tx=0).stop 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 shutdown_connections 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd7f80a3d10 0x7fd7f80a61d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd810107b20 0x7fd81019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8101051f0 0x7fd8101986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 --2- 172.21.15.16:0/2797785928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8101028c0 0x7fd810198190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 >> 172.21.15.16:0/2797785928 conn(0x7fd8100fc2b0 msgr2=0x7fd8101032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 shutdown_connections 2024-04-13T03:34:48.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:34:48.251+0000 7fd815e00640 1 -- 172.21.15.16:0/2797785928 wait complete. 2024-04-13T03:34:48.272 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-13T03:34:48.354 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-13T03:34:48.364 INFO:teuthology.run_tasks:Running task workunit... 2024-04-13T03:34:48.393 INFO:tasks.workunit:Pulling workunits from ref 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:34:48.393 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-13T03:34:48.393 INFO:tasks.workunit:timeout=3h 2024-04-13T03:34:48.394 INFO:tasks.workunit:cleanup=True 2024-04-13T03:34:48.394 DEBUG:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-13T03:34:48.440 INFO:teuthology.orchestra.run.smithi016.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-13T03:34:48.441 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-13T03:34:48.441 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-04-13T03:34:48.441 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-13T03:34:48.441 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-13T03:34:48.441 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-13 03:32:08.148317799 +0000 2024-04-13T03:34:48.441 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-13 03:32:08.148317799 +0000 2024-04-13T03:34:48.441 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-13 03:34:43.896028217 +0000 2024-04-13T03:34:48.441 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-13 03:32:08.148317799 +0000 2024-04-13T03:34:48.441 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-13T03:34:48.442 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-13T03:34:48.550 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:48 smithi016 ceph-mon[31772]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:48.550 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:48 smithi016 ceph-mon[31772]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:48.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:48 smithi016 ceph-mon[31772]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-13T03:34:48.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:48 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:48 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:48 smithi016 ceph-mon[31772]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:48.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:48 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:48 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.570 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-13T03:34:48.617 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-13T03:34:48.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:48 smithi119 ceph-mon[35835]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:48.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:48 smithi119 ceph-mon[35835]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:48.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:48 smithi119 ceph-mon[35835]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-13T03:34:48.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:48 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:48 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:48 smithi119 ceph-mon[35835]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:48.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:48 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:48 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:48 smithi188 ceph-mon[36398]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:48.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:48 smithi188 ceph-mon[36398]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:48.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:48 smithi188 ceph-mon[36398]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-13T03:34:48.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:48 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:48 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:48 smithi188 ceph-mon[36398]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:48.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:48 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:48.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:48 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='client.25556 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:49 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='client.25556 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:49.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.857 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:49 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='client.25556 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:34:49.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:49.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:49.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:49 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:50.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:50 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi188 to 174.1M 2024-04-13T03:34:50.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:50 smithi119 ceph-mon[35835]: Unable to set osd_memory_target on smithi188 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:34:50.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:50 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi119 to 174.1M 2024-04-13T03:34:50.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:50 smithi119 ceph-mon[35835]: Unable to set osd_memory_target on smithi119 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:34:50.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:50 smithi119 ceph-mon[35835]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:50.822 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:50 smithi119 ceph-mon[35835]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-13T03:34:50.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:50 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi188 to 174.1M 2024-04-13T03:34:50.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:50 smithi016 ceph-mon[31772]: Unable to set osd_memory_target on smithi188 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:34:50.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:50 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi119 to 174.1M 2024-04-13T03:34:50.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:50 smithi016 ceph-mon[31772]: Unable to set osd_memory_target on smithi119 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:34:50.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:50 smithi016 ceph-mon[31772]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:50.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:50 smithi016 ceph-mon[31772]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-13T03:34:50.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:50 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi188 to 174.1M 2024-04-13T03:34:50.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:50 smithi188 ceph-mon[36398]: Unable to set osd_memory_target on smithi188 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:34:50.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:50 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi119 to 174.1M 2024-04-13T03:34:50.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:50 smithi188 ceph-mon[36398]: Unable to set osd_memory_target on smithi119 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:34:50.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:50 smithi188 ceph-mon[36398]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:50.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:50 smithi188 ceph-mon[36398]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-13T03:34:52.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:52.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:52.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi016 to 177.5M 2024-04-13T03:34:52.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: Unable to set osd_memory_target on smithi016 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-13T03:34:52.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:52.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:52.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-13T03:34:52.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-13T03:34:52.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-13T03:34:52.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:34:52.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:34:52.324 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:52.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:34:52.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:51 smithi119 ceph-mon[35835]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:34:52.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:52.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:52.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi016 to 177.5M 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: Unable to set osd_memory_target on smithi016 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:34:52.357 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:51 smithi016 ceph-mon[31772]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:34:52.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:52.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:52.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi016 to 177.5M 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: Unable to set osd_memory_target on smithi016 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:34:52.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:52.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:34:52.376 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:51 smithi188 ceph-mon[36398]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:34:53.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:53 smithi119 ceph-mon[35835]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:34:53.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:53 smithi119 ceph-mon[35835]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:34:53.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:53 smithi119 ceph-mon[35835]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:34:53.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:53 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:53 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:53 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:53 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:53 smithi119 ceph-mon[35835]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:34:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:53 smithi016 ceph-mon[31772]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:34:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:53 smithi016 ceph-mon[31772]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:34:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:53 smithi016 ceph-mon[31772]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:34:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:53 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:53 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:53 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:53 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:53 smithi016 ceph-mon[31772]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:34:53.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:53 smithi188 ceph-mon[36398]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-13T03:34:53.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:53 smithi188 ceph-mon[36398]: Updating smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:34:53.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:53 smithi188 ceph-mon[36398]: Updating smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:34:53.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:53 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:53 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:53 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:53 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:53.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:53 smithi188 ceph-mon[36398]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-13T03:34:54.554 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:54 smithi188 ceph-mon[36398]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:34:54.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:54 smithi188 ceph-mon[36398]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 79 op/s 2024-04-13T03:34:54.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:54 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:54.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:54 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:54.555 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:54 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:54.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:54 smithi119 ceph-mon[35835]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:34:54.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:54 smithi119 ceph-mon[35835]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 79 op/s 2024-04-13T03:34:54.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:54 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:54.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:54 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:54.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:54 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:54 smithi016 ceph-mon[31772]: Updating smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-13T03:34:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:54 smithi016 ceph-mon[31772]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 79 op/s 2024-04-13T03:34:54.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:54 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:54.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:54 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:54.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:54 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:55.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:55 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:55.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:55 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:55.491 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:55 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:55.492 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:55 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:55.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:55 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:55.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:55 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:55.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:55 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:55.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:55 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:55.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:55 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:55.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:55 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:55.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:55 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:55.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:55 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:34:56.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:56 smithi119 ceph-mon[35835]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 61 op/s 2024-04-13T03:34:56.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:56 smithi016 ceph-mon[31772]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 61 op/s 2024-04-13T03:34:56.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:56 smithi188 ceph-mon[36398]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 61 op/s 2024-04-13T03:34:58.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:34:58 smithi119 ceph-mon[35835]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-04-13T03:34:58.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:34:58 smithi016 ceph-mon[31772]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-04-13T03:34:58.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:34:58 smithi188 ceph-mon[36398]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-04-13T03:35:00.468 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:35:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:35:00.207+0000 7fed74200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:35:00.469 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:35:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:35:00.207+0000 7fed74200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:35:00.469 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:35:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:35:00.207+0000 7fed74200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:35:00.469 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:35:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:35:00.207+0000 7fed74200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:35:00.469 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:35:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:35:00.207+0000 7fed74200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-13T03:35:00.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:00 smithi119 ceph-mon[35835]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-13T03:35:00.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:00 smithi016 ceph-mon[31772]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-13T03:35:00.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:00 smithi188 ceph-mon[36398]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-13T03:35:02.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:02 smithi119 ceph-mon[35835]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-13T03:35:02.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:02 smithi119 ceph-mon[35835]: mgrmap e25: x(active, since 16s), standbys: y, z 2024-04-13T03:35:02.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:02 smithi016 ceph-mon[31772]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-13T03:35:02.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:02 smithi016 ceph-mon[31772]: mgrmap e25: x(active, since 16s), standbys: y, z 2024-04-13T03:35:02.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:02 smithi188 ceph-mon[36398]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-13T03:35:02.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:02 smithi188 ceph-mon[36398]: mgrmap e25: x(active, since 16s), standbys: y, z 2024-04-13T03:35:04.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:04 smithi119 ceph-mon[35835]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 100 KiB/s wr, 70 op/s 2024-04-13T03:35:04.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:04 smithi016 ceph-mon[31772]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 100 KiB/s wr, 70 op/s 2024-04-13T03:35:04.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:04 smithi188 ceph-mon[36398]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 100 KiB/s wr, 70 op/s 2024-04-13T03:35:06.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:06 smithi119 ceph-mon[35835]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-13T03:35:06.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:06 smithi016 ceph-mon[31772]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-13T03:35:06.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:06 smithi188 ceph-mon[36398]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-13T03:35:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:08 smithi119 ceph-mon[35835]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-13T03:35:08.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:08 smithi016 ceph-mon[31772]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-13T03:35:08.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:08 smithi188 ceph-mon[36398]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-13T03:35:10.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:10 smithi119 ceph-mon[35835]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-13T03:35:10.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:10 smithi016 ceph-mon[31772]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-13T03:35:10.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:10 smithi188 ceph-mon[36398]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-13T03:35:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:12 smithi119 ceph-mon[35835]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-04-13T03:35:12.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:12 smithi016 ceph-mon[31772]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-04-13T03:35:12.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:12 smithi188 ceph-mon[36398]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-04-13T03:35:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:14 smithi119 ceph-mon[35835]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-13T03:35:14.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:14 smithi016 ceph-mon[31772]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-13T03:35:14.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:14 smithi188 ceph-mon[36398]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-13T03:35:16.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:16 smithi119 ceph-mon[35835]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-13T03:35:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:16 smithi016 ceph-mon[31772]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-13T03:35:16.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:16 smithi188 ceph-mon[36398]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-13T03:35:18.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:18 smithi119 ceph-mon[35835]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-13T03:35:18.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:18 smithi016 ceph-mon[31772]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-13T03:35:18.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:18 smithi188 ceph-mon[36398]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-13T03:35:20.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:20 smithi119 ceph-mon[35835]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:20.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:20 smithi016 ceph-mon[31772]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:20.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:20 smithi188 ceph-mon[36398]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:22.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:22 smithi119 ceph-mon[35835]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-13T03:35:22.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:22 smithi016 ceph-mon[31772]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-13T03:35:22.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:22 smithi188 ceph-mon[36398]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-13T03:35:24.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:24 smithi119 ceph-mon[35835]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 11 op/s 2024-04-13T03:35:24.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:24 smithi016 ceph-mon[31772]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 11 op/s 2024-04-13T03:35:24.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:24 smithi188 ceph-mon[36398]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 11 op/s 2024-04-13T03:35:26.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:26 smithi119 ceph-mon[35835]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:26.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:26 smithi016 ceph-mon[31772]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:26.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:26 smithi188 ceph-mon[36398]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:28.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:28 smithi119 ceph-mon[35835]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-13T03:35:28.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:28 smithi016 ceph-mon[31772]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-13T03:35:28.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:28 smithi188 ceph-mon[36398]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-13T03:35:30.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:30 smithi119 ceph-mon[35835]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-13T03:35:30.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:30 smithi016 ceph-mon[31772]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-13T03:35:30.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:30 smithi188 ceph-mon[36398]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-13T03:35:32.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:32 smithi119 ceph-mon[35835]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-13T03:35:32.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:32 smithi016 ceph-mon[31772]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-13T03:35:32.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:32 smithi188 ceph-mon[36398]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-13T03:35:34.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:34 smithi119 ceph-mon[35835]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-04-13T03:35:34.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:34 smithi016 ceph-mon[31772]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-04-13T03:35:34.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:34 smithi188 ceph-mon[36398]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-04-13T03:35:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:36 smithi016 ceph-mon[31772]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-13T03:35:36.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:36 smithi188 ceph-mon[36398]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-13T03:35:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:36 smithi119 ceph-mon[35835]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-13T03:35:38.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:38 smithi016 ceph-mon[31772]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-13T03:35:38.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:38 smithi188 ceph-mon[36398]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-13T03:35:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:38 smithi119 ceph-mon[35835]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-04-13T03:35:40.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:40 smithi016 ceph-mon[31772]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:40.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:40 smithi188 ceph-mon[36398]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:40 smithi119 ceph-mon[35835]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:42.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:42 smithi016 ceph-mon[31772]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-13T03:35:42.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:42 smithi188 ceph-mon[36398]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-13T03:35:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:42 smithi119 ceph-mon[35835]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-13T03:35:44.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:44 smithi016 ceph-mon[31772]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-13T03:35:44.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:44 smithi188 ceph-mon[36398]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-13T03:35:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:44 smithi119 ceph-mon[35835]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-13T03:35:46.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:46 smithi016 ceph-mon[31772]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:46.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:46 smithi188 ceph-mon[36398]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:46 smithi119 ceph-mon[35835]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:48.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:48 smithi016 ceph-mon[31772]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-13T03:35:48.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:48 smithi188 ceph-mon[36398]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-13T03:35:49.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:48 smithi119 ceph-mon[35835]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-13T03:35:50.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:50 smithi016 ceph-mon[31772]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:50.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:50 smithi188 ceph-mon[36398]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:50 smithi119 ceph-mon[35835]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:52.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:52 smithi016 ceph-mon[31772]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-13T03:35:52.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:52 smithi188 ceph-mon[36398]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-13T03:35:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:52 smithi119 ceph-mon[35835]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-13T03:35:54.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:54 smithi188 ceph-mon[36398]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-13T03:35:55.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:54 smithi119 ceph-mon[35835]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-13T03:35:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:54 smithi016 ceph-mon[31772]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-04-13T03:35:55.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:55 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:35:56.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:55 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:35:56.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:55 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:35:56.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:56 smithi188 ceph-mon[36398]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:56.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:56 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:35:56.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:56 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:35:56.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:56 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:35:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:56 smithi119 ceph-mon[35835]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:56 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:35:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:56 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:35:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:56 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:35:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:56 smithi016 ceph-mon[31772]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:35:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:56 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:35:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:56 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:35:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:56 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:35:58.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:35:58 smithi188 ceph-mon[36398]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-13T03:35:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:35:58 smithi119 ceph-mon[35835]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-13T03:35:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:35:58 smithi016 ceph-mon[31772]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-13T03:36:00.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:00 smithi188 ceph-mon[36398]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:36:01.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:00 smithi119 ceph-mon[35835]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:36:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:00 smithi016 ceph-mon[31772]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-13T03:36:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:02 smithi119 ceph-mon[35835]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-13T03:36:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:02 smithi016 ceph-mon[31772]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-13T03:36:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:02 smithi188 ceph-mon[36398]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-13T03:36:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:04 smithi119 ceph-mon[35835]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-13T03:36:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:04 smithi016 ceph-mon[31772]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-13T03:36:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:04 smithi188 ceph-mon[36398]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-13T03:36:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:06 smithi119 ceph-mon[35835]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-13T03:36:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:06 smithi016 ceph-mon[31772]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-13T03:36:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:06 smithi188 ceph-mon[36398]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-13T03:36:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:08 smithi119 ceph-mon[35835]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-13T03:36:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:08 smithi016 ceph-mon[31772]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-13T03:36:09.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:08 smithi188 ceph-mon[36398]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-13T03:36:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:10 smithi119 ceph-mon[35835]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-13T03:36:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:10 smithi016 ceph-mon[31772]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-13T03:36:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:10 smithi188 ceph-mon[36398]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-13T03:36:12.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:12 smithi016 ceph-mon[31772]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-13T03:36:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:12 smithi119 ceph-mon[35835]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-13T03:36:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:12 smithi188 ceph-mon[36398]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-13T03:36:15.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:14 smithi119 ceph-mon[35835]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-04-13T03:36:15.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:14 smithi016 ceph-mon[31772]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-04-13T03:36:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:14 smithi188 ceph-mon[36398]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 21 op/s 2024-04-13T03:36:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:16 smithi119 ceph-mon[35835]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-13T03:36:17.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:16 smithi016 ceph-mon[31772]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-13T03:36:17.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:16 smithi188 ceph-mon[36398]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-13T03:36:17.423 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 58% (7179/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 96% (11790/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-13T03:36:18.213 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to '218a691417ef57c5d22a8d5cf3c5f41f7b542838'. 2024-04-13T03:36:18.213 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-13T03:36:18.213 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-13T03:36:18.214 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-04-13T03:36:18.215 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at 218a691417e Merge PR #52623 into wip-vshankar-testing-20240412.042930 2024-04-13T03:36:18.228 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-13T03:36:18.335 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-13T03:36:18.342 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-13T03:36:18.343 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-13T03:36:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:18 smithi119 ceph-mon[35835]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.3 KiB/s wr, 14 op/s 2024-04-13T03:36:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:18 smithi119 ceph-mon[35835]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-13T03:36:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:18 smithi016 ceph-mon[31772]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.3 KiB/s wr, 14 op/s 2024-04-13T03:36:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:18 smithi016 ceph-mon[31772]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-13T03:36:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:18 smithi188 ceph-mon[36398]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.3 KiB/s wr, 14 op/s 2024-04-13T03:36:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:18 smithi188 ceph-mon[36398]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-13T03:36:19.261 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-13T03:36:19.417 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-13T03:36:19.505 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-13T03:36:19.512 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-13T03:36:19.512 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-13T03:36:19.595 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-13T03:36:19.604 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:36:19.604 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-13T03:36:19.658 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-13T03:36:19.660 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-13T03:36:19.660 DEBUG:teuthology.orchestra.run.smithi016:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-13T03:36:19.737 INFO:tasks.workunit.client.0.smithi016.stderr:+ dbench 1 2024-04-13T03:36:19.753 INFO:tasks.workunit.client.0.smithi016.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-13T03:36:19.753 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-04-13T03:36:19.753 INFO:tasks.workunit.client.0.smithi016.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-13T03:36:19.762 INFO:tasks.workunit.client.0.smithi016.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-13T03:36:19.862 INFO:tasks.workunit.client.0.smithi016.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-13T03:36:19.862 INFO:tasks.workunit.client.0.smithi016.stdout:releasing clients 2024-04-13T03:36:20.862 INFO:tasks.workunit.client.0.smithi016.stdout: 1 1481 26.04 MB/sec warmup 1 sec latency 12.223 ms 2024-04-13T03:36:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:20 smithi119 ceph-mon[35835]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-13T03:36:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:20 smithi016 ceph-mon[31772]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-13T03:36:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:20 smithi188 ceph-mon[36398]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-13T03:36:21.862 INFO:tasks.workunit.client.0.smithi016.stdout: 1 3868 19.88 MB/sec warmup 2 sec latency 13.982 ms 2024-04-13T03:36:22.863 INFO:tasks.workunit.client.0.smithi016.stdout: 1 5622 15.25 MB/sec warmup 3 sec latency 15.822 ms 2024-04-13T03:36:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:22 smithi119 ceph-mon[35835]: pgmap v51: 129 pgs: 129 active+clean; 2.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 129 KiB/s wr, 31 op/s 2024-04-13T03:36:23.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:22 smithi016 ceph-mon[31772]: pgmap v51: 129 pgs: 129 active+clean; 2.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 129 KiB/s wr, 31 op/s 2024-04-13T03:36:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:22 smithi188 ceph-mon[36398]: pgmap v51: 129 pgs: 129 active+clean; 2.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 129 KiB/s wr, 31 op/s 2024-04-13T03:36:23.863 INFO:tasks.workunit.client.0.smithi016.stdout: 1 7543 14.79 MB/sec warmup 4 sec latency 12.996 ms 2024-04-13T03:36:24.863 INFO:tasks.workunit.client.0.smithi016.stdout: 1 9510 13.25 MB/sec warmup 5 sec latency 18.430 ms 2024-04-13T03:36:25.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:24 smithi119 ceph-mon[35835]: pgmap v52: 129 pgs: 129 active+clean; 2.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 141 KiB/s wr, 27 op/s 2024-04-13T03:36:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:24 smithi016 ceph-mon[31772]: pgmap v52: 129 pgs: 129 active+clean; 2.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 141 KiB/s wr, 27 op/s 2024-04-13T03:36:25.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:24 smithi188 ceph-mon[36398]: pgmap v52: 129 pgs: 129 active+clean; 2.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 141 KiB/s wr, 27 op/s 2024-04-13T03:36:25.863 INFO:tasks.workunit.client.0.smithi016.stdout: 1 11242 13.05 MB/sec warmup 6 sec latency 10.966 ms 2024-04-13T03:36:26.863 INFO:tasks.workunit.client.0.smithi016.stdout: 1 13371 12.75 MB/sec warmup 7 sec latency 12.968 ms 2024-04-13T03:36:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:26 smithi119 ceph-mon[35835]: pgmap v53: 129 pgs: 129 active+clean; 2.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 141 KiB/s wr, 27 op/s 2024-04-13T03:36:27.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:26 smithi016 ceph-mon[31772]: pgmap v53: 129 pgs: 129 active+clean; 2.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 141 KiB/s wr, 27 op/s 2024-04-13T03:36:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:26 smithi188 ceph-mon[36398]: pgmap v53: 129 pgs: 129 active+clean; 2.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 141 KiB/s wr, 27 op/s 2024-04-13T03:36:27.863 INFO:tasks.workunit.client.0.smithi016.stdout: 1 14917 12.21 MB/sec warmup 8 sec latency 41.509 ms 2024-04-13T03:36:28.863 INFO:tasks.workunit.client.0.smithi016.stdout: 1 17199 12.22 MB/sec warmup 9 sec latency 10.922 ms 2024-04-13T03:36:29.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:28 smithi119 ceph-mon[35835]: pgmap v54: 129 pgs: 129 active+clean; 7.7 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 635 KiB/s wr, 86 op/s 2024-04-13T03:36:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:28 smithi016 ceph-mon[31772]: pgmap v54: 129 pgs: 129 active+clean; 7.7 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 635 KiB/s wr, 86 op/s 2024-04-13T03:36:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:28 smithi188 ceph-mon[36398]: pgmap v54: 129 pgs: 129 active+clean; 7.7 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 635 KiB/s wr, 86 op/s 2024-04-13T03:36:29.863 INFO:tasks.workunit.client.0.smithi016.stdout: 1 18736 11.75 MB/sec warmup 10 sec latency 12.515 ms 2024-04-13T03:36:30.863 INFO:tasks.workunit.client.0.smithi016.stdout: 1 21333 12.19 MB/sec warmup 11 sec latency 8.880 ms 2024-04-13T03:36:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:30 smithi119 ceph-mon[35835]: pgmap v55: 129 pgs: 129 active+clean; 7.7 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 633 KiB/s wr, 80 op/s 2024-04-13T03:36:31.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:30 smithi016 ceph-mon[31772]: pgmap v55: 129 pgs: 129 active+clean; 7.7 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 633 KiB/s wr, 80 op/s 2024-04-13T03:36:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:30 smithi188 ceph-mon[36398]: pgmap v55: 129 pgs: 129 active+clean; 7.7 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 633 KiB/s wr, 80 op/s 2024-04-13T03:36:31.865 INFO:tasks.workunit.client.0.smithi016.stdout: 1 22777 11.73 MB/sec warmup 12 sec latency 13.145 ms 2024-04-13T03:36:32.864 INFO:tasks.workunit.client.0.smithi016.stdout: 1 25127 11.89 MB/sec warmup 13 sec latency 11.153 ms 2024-04-13T03:36:33.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:32 smithi119 ceph-mon[35835]: pgmap v56: 129 pgs: 129 active+clean; 12 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.1 MiB/s wr, 139 op/s 2024-04-13T03:36:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:32 smithi016 ceph-mon[31772]: pgmap v56: 129 pgs: 129 active+clean; 12 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.1 MiB/s wr, 139 op/s 2024-04-13T03:36:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:32 smithi188 ceph-mon[36398]: pgmap v56: 129 pgs: 129 active+clean; 12 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.1 MiB/s wr, 139 op/s 2024-04-13T03:36:33.864 INFO:tasks.workunit.client.0.smithi016.stdout: 1 26484 11.43 MB/sec warmup 14 sec latency 37.791 ms 2024-04-13T03:36:34.865 INFO:tasks.workunit.client.0.smithi016.stdout: 1 28761 11.60 MB/sec warmup 15 sec latency 13.372 ms 2024-04-13T03:36:35.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:34 smithi119 ceph-mon[35835]: pgmap v57: 129 pgs: 129 active+clean; 12 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 977 KiB/s wr, 117 op/s 2024-04-13T03:36:35.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:34 smithi016 ceph-mon[31772]: pgmap v57: 129 pgs: 129 active+clean; 12 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 977 KiB/s wr, 117 op/s 2024-04-13T03:36:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:34 smithi188 ceph-mon[36398]: pgmap v57: 129 pgs: 129 active+clean; 12 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 977 KiB/s wr, 117 op/s 2024-04-13T03:36:35.865 INFO:tasks.workunit.client.0.smithi016.stdout: 1 30332 11.22 MB/sec warmup 16 sec latency 15.974 ms 2024-04-13T03:36:36.865 INFO:tasks.workunit.client.0.smithi016.stdout: 1 32354 11.36 MB/sec warmup 17 sec latency 9.583 ms 2024-04-13T03:36:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:36 smithi119 ceph-mon[35835]: pgmap v58: 129 pgs: 129 active+clean; 12 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 966 KiB/s wr, 117 op/s 2024-04-13T03:36:37.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:36 smithi016 ceph-mon[31772]: pgmap v58: 129 pgs: 129 active+clean; 12 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 966 KiB/s wr, 117 op/s 2024-04-13T03:36:37.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:36 smithi188 ceph-mon[36398]: pgmap v58: 129 pgs: 129 active+clean; 12 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 966 KiB/s wr, 117 op/s 2024-04-13T03:36:37.865 INFO:tasks.workunit.client.0.smithi016.stdout: 1 34105 11.04 MB/sec warmup 18 sec latency 19.124 ms 2024-04-13T03:36:38.865 INFO:tasks.workunit.client.0.smithi016.stdout: 1 35890 11.17 MB/sec warmup 19 sec latency 16.252 ms 2024-04-13T03:36:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:38 smithi119 ceph-mon[35835]: pgmap v59: 129 pgs: 129 active+clean; 16 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.4 MiB/s wr, 178 op/s 2024-04-13T03:36:39.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:38 smithi016 ceph-mon[31772]: pgmap v59: 129 pgs: 129 active+clean; 16 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.4 MiB/s wr, 178 op/s 2024-04-13T03:36:39.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:38 smithi188 ceph-mon[36398]: pgmap v59: 129 pgs: 129 active+clean; 16 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 1.4 MiB/s wr, 178 op/s 2024-04-13T03:36:39.865 INFO:tasks.workunit.client.0.smithi016.stdout: 1 37854 10.96 MB/sec warmup 20 sec latency 12.689 ms 2024-04-13T03:36:40.865 INFO:tasks.workunit.client.0.smithi016.stdout: 1 39518 11.01 MB/sec warmup 21 sec latency 9.616 ms 2024-04-13T03:36:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:40 smithi119 ceph-mon[35835]: pgmap v60: 129 pgs: 129 active+clean; 16 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 956 KiB/s wr, 120 op/s 2024-04-13T03:36:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:40 smithi016 ceph-mon[31772]: pgmap v60: 129 pgs: 129 active+clean; 16 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 956 KiB/s wr, 120 op/s 2024-04-13T03:36:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:40 smithi188 ceph-mon[36398]: pgmap v60: 129 pgs: 129 active+clean; 16 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 956 KiB/s wr, 120 op/s 2024-04-13T03:36:41.866 INFO:tasks.workunit.client.0.smithi016.stdout: 1 41358 10.82 MB/sec warmup 22 sec latency 18.227 ms 2024-04-13T03:36:42.866 INFO:tasks.workunit.client.0.smithi016.stdout: 1 42839 10.87 MB/sec warmup 23 sec latency 114.968 ms 2024-04-13T03:36:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:42 smithi119 ceph-mon[35835]: pgmap v61: 129 pgs: 129 active+clean; 21 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-04-13T03:36:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:42 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:36:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:42 smithi016 ceph-mon[31772]: pgmap v61: 129 pgs: 129 active+clean; 21 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-04-13T03:36:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:42 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:36:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:42 smithi188 ceph-mon[36398]: pgmap v61: 129 pgs: 129 active+clean; 21 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-04-13T03:36:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:42 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:36:43.866 INFO:tasks.workunit.client.0.smithi016.stdout: 1 42839 10.42 MB/sec warmup 24 sec latency 1115.128 ms 2024-04-13T03:36:44.866 INFO:tasks.workunit.client.0.smithi016.stdout: 1 42839 10.00 MB/sec warmup 25 sec latency 2115.291 ms 2024-04-13T03:36:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:44 smithi119 ceph-mon[35835]: pgmap v62: 129 pgs: 129 active+clean; 21 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 949 KiB/s wr, 118 op/s 2024-04-13T03:36:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:44 smithi016 ceph-mon[31772]: pgmap v62: 129 pgs: 129 active+clean; 21 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 949 KiB/s wr, 118 op/s 2024-04-13T03:36:45.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:44 smithi188 ceph-mon[36398]: pgmap v62: 129 pgs: 129 active+clean; 21 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 949 KiB/s wr, 118 op/s 2024-04-13T03:36:45.866 INFO:tasks.workunit.client.0.smithi016.stdout: 1 42839 9.62 MB/sec warmup 26 sec latency 3115.406 ms 2024-04-13T03:36:46.866 INFO:tasks.workunit.client.0.smithi016.stdout: 1 42877 9.26 MB/sec warmup 27 sec latency 4002.266 ms 2024-04-13T03:36:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:46 smithi119 ceph-mon[35835]: pgmap v63: 129 pgs: 129 active+clean; 21 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 949 KiB/s wr, 118 op/s 2024-04-13T03:36:47.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:46 smithi016 ceph-mon[31772]: pgmap v63: 129 pgs: 129 active+clean; 21 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 949 KiB/s wr, 118 op/s 2024-04-13T03:36:47.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:46 smithi188 ceph-mon[36398]: pgmap v63: 129 pgs: 129 active+clean; 21 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 949 KiB/s wr, 118 op/s 2024-04-13T03:36:47.867 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43320 8.95 MB/sec warmup 28 sec latency 351.084 ms 2024-04-13T03:36:48.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:47 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:36:48.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:47 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:36:48.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:47 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:36:48.867 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43320 8.64 MB/sec warmup 29 sec latency 1351.255 ms 2024-04-13T03:36:49.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:48 smithi119 ceph-mon[35835]: pgmap v64: 129 pgs: 129 active+clean; 21 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-04-13T03:36:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:48 smithi016 ceph-mon[31772]: pgmap v64: 129 pgs: 129 active+clean; 21 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-04-13T03:36:49.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:48 smithi188 ceph-mon[36398]: pgmap v64: 129 pgs: 129 active+clean; 21 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-04-13T03:36:49.867 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43320 8.36 MB/sec warmup 30 sec latency 2351.418 ms 2024-04-13T03:36:50.867 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43320 8.09 MB/sec warmup 31 sec latency 3351.572 ms 2024-04-13T03:36:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:50 smithi119 ceph-mon[35835]: pgmap v65: 129 pgs: 129 active+clean; 21 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 559 KiB/s wr, 78 op/s 2024-04-13T03:36:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:50 smithi016 ceph-mon[31772]: pgmap v65: 129 pgs: 129 active+clean; 21 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 559 KiB/s wr, 78 op/s 2024-04-13T03:36:51.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:50 smithi188 ceph-mon[36398]: pgmap v65: 129 pgs: 129 active+clean; 21 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 559 KiB/s wr, 78 op/s 2024-04-13T03:36:51.867 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43424 7.84 MB/sec warmup 32 sec latency 4243.358 ms 2024-04-13T03:36:52.867 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43576 7.61 MB/sec warmup 33 sec latency 873.127 ms 2024-04-13T03:36:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:52 smithi119 ceph-mon[35835]: pgmap v66: 129 pgs: 129 active+clean; 42 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 2.3 MiB/s wr, 101 op/s 2024-04-13T03:36:53.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:52 smithi016 ceph-mon[31772]: pgmap v66: 129 pgs: 129 active+clean; 42 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 2.3 MiB/s wr, 101 op/s 2024-04-13T03:36:53.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:52 smithi188 ceph-mon[36398]: pgmap v66: 129 pgs: 129 active+clean; 42 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 2.3 MiB/s wr, 101 op/s 2024-04-13T03:36:53.867 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43576 7.39 MB/sec warmup 34 sec latency 1873.218 ms 2024-04-13T03:36:54.868 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43576 7.18 MB/sec warmup 35 sec latency 2873.376 ms 2024-04-13T03:36:55.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:54 smithi119 ceph-mon[35835]: pgmap v67: 129 pgs: 129 active+clean; 42 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.8 MiB/s wr, 44 op/s 2024-04-13T03:36:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:54 smithi016 ceph-mon[31772]: pgmap v67: 129 pgs: 129 active+clean; 42 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.8 MiB/s wr, 44 op/s 2024-04-13T03:36:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:54 smithi188 ceph-mon[36398]: pgmap v67: 129 pgs: 129 active+clean; 42 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.8 MiB/s wr, 44 op/s 2024-04-13T03:36:55.868 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43576 6.98 MB/sec warmup 36 sec latency 3873.564 ms 2024-04-13T03:36:56.869 INFO:tasks.workunit.client.0.smithi016.stdout: 1 43665 6.82 MB/sec warmup 37 sec latency 4775.388 ms 2024-04-13T03:36:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:56 smithi119 ceph-mon[35835]: pgmap v68: 129 pgs: 129 active+clean; 42 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.8 MiB/s wr, 44 op/s 2024-04-13T03:36:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:56 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:36:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:56 smithi016 ceph-mon[31772]: pgmap v68: 129 pgs: 129 active+clean; 42 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.8 MiB/s wr, 44 op/s 2024-04-13T03:36:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:56 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:36:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:56 smithi188 ceph-mon[36398]: pgmap v68: 129 pgs: 129 active+clean; 42 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.8 MiB/s wr, 44 op/s 2024-04-13T03:36:57.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:56 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:36:57.869 INFO:tasks.workunit.client.0.smithi016.stdout: 1 45573 6.92 MB/sec warmup 38 sec latency 12.830 ms 2024-04-13T03:36:58.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:57 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:36:58.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:57 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:36:58.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:57 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:36:58.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:57 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:36:58.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:57 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:36:58.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:57 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:36:58.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:57 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:36:58.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:57 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:36:58.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:57 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:36:58.869 INFO:tasks.workunit.client.0.smithi016.stdout: 1 46303 6.88 MB/sec warmup 39 sec latency 441.912 ms 2024-04-13T03:36:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:36:58 smithi119 ceph-mon[35835]: pgmap v69: 129 pgs: 129 active+clean; 50 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 2.6 MiB/s wr, 70 op/s 2024-04-13T03:36:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:36:58 smithi016 ceph-mon[31772]: pgmap v69: 129 pgs: 129 active+clean; 50 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 2.6 MiB/s wr, 70 op/s 2024-04-13T03:36:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:36:58 smithi188 ceph-mon[36398]: pgmap v69: 129 pgs: 129 active+clean; 50 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 2.6 MiB/s wr, 70 op/s 2024-04-13T03:36:59.869 INFO:tasks.workunit.client.0.smithi016.stdout: 1 46400 6.73 MB/sec warmup 40 sec latency 1289.721 ms 2024-04-13T03:37:00.869 INFO:tasks.workunit.client.0.smithi016.stdout: 1 46814 6.58 MB/sec warmup 41 sec latency 313.690 ms 2024-04-13T03:37:01.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:00 smithi119 ceph-mon[35835]: pgmap v70: 129 pgs: 129 active+clean; 50 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.5 MiB/s wr, 49 op/s 2024-04-13T03:37:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:00 smithi016 ceph-mon[31772]: pgmap v70: 129 pgs: 129 active+clean; 50 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.5 MiB/s wr, 49 op/s 2024-04-13T03:37:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:00 smithi188 ceph-mon[36398]: pgmap v70: 129 pgs: 129 active+clean; 50 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 2.5 MiB/s wr, 49 op/s 2024-04-13T03:37:01.869 INFO:tasks.workunit.client.0.smithi016.stdout: 1 46880 6.43 MB/sec warmup 42 sec latency 1222.170 ms 2024-04-13T03:37:02.869 INFO:tasks.workunit.client.0.smithi016.stdout: 1 47178 6.31 MB/sec warmup 43 sec latency 86.146 ms 2024-04-13T03:37:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:02 smithi119 ceph-mon[35835]: pgmap v71: 129 pgs: 129 active+clean; 50 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 3.3 MiB/s wr, 125 op/s 2024-04-13T03:37:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:02 smithi016 ceph-mon[31772]: pgmap v71: 129 pgs: 129 active+clean; 50 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 3.3 MiB/s wr, 125 op/s 2024-04-13T03:37:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:02 smithi188 ceph-mon[36398]: pgmap v71: 129 pgs: 129 active+clean; 50 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 3.3 MiB/s wr, 125 op/s 2024-04-13T03:37:03.869 INFO:tasks.workunit.client.0.smithi016.stdout: 1 48549 6.29 MB/sec warmup 44 sec latency 74.717 ms 2024-04-13T03:37:04.869 INFO:tasks.workunit.client.0.smithi016.stdout: 1 49839 6.39 MB/sec warmup 45 sec latency 16.576 ms 2024-04-13T03:37:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:04 smithi119 ceph-mon[35835]: pgmap v72: 129 pgs: 129 active+clean; 50 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-13T03:37:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:04 smithi016 ceph-mon[31772]: pgmap v72: 129 pgs: 129 active+clean; 50 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-13T03:37:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:04 smithi188 ceph-mon[36398]: pgmap v72: 129 pgs: 129 active+clean; 50 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-13T03:37:05.870 INFO:tasks.workunit.client.0.smithi016.stdout: 1 50401 6.28 MB/sec warmup 46 sec latency 65.832 ms 2024-04-13T03:37:06.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:05 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:06.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:05 smithi016 ceph-mon[31772]: pgmap v73: 129 pgs: 129 active+clean; 50 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-13T03:37:06.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:05 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:06.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:05 smithi188 ceph-mon[36398]: pgmap v73: 129 pgs: 129 active+clean; 50 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-13T03:37:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:05 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:06.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:05 smithi119 ceph-mon[35835]: pgmap v73: 129 pgs: 129 active+clean; 50 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-13T03:37:06.870 INFO:tasks.workunit.client.0.smithi016.stdout: 1 50587 6.16 MB/sec warmup 47 sec latency 783.348 ms 2024-04-13T03:37:07.871 INFO:tasks.workunit.client.0.smithi016.stdout: 1 52026 6.16 MB/sec warmup 48 sec latency 214.313 ms 2024-04-13T03:37:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:08 smithi119 ceph-mon[35835]: pgmap v74: 129 pgs: 129 active+clean; 51 MiB data, 474 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-04-13T03:37:08.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:08 smithi016 ceph-mon[31772]: pgmap v74: 129 pgs: 129 active+clean; 51 MiB data, 474 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-04-13T03:37:08.870 INFO:tasks.workunit.client.0.smithi016.stdout: 1 52080 6.03 MB/sec warmup 49 sec latency 705.846 ms 2024-04-13T03:37:08.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:08 smithi188 ceph-mon[36398]: pgmap v74: 129 pgs: 129 active+clean; 51 MiB data, 474 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-04-13T03:37:09.870 INFO:tasks.workunit.client.0.smithi016.stdout: 1 52080 5.91 MB/sec warmup 50 sec latency 1705.973 ms 2024-04-13T03:37:10.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:10 smithi119 ceph-mon[35835]: pgmap v75: 129 pgs: 129 active+clean; 51 MiB data, 474 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.8 MiB/s wr, 166 op/s 2024-04-13T03:37:10.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:10 smithi016 ceph-mon[31772]: pgmap v75: 129 pgs: 129 active+clean; 51 MiB data, 474 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.8 MiB/s wr, 166 op/s 2024-04-13T03:37:10.870 INFO:tasks.workunit.client.0.smithi016.stdout: 1 52080 5.80 MB/sec warmup 51 sec latency 2706.143 ms 2024-04-13T03:37:10.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:10 smithi188 ceph-mon[36398]: pgmap v75: 129 pgs: 129 active+clean; 51 MiB data, 474 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 1.8 MiB/s wr, 166 op/s 2024-04-13T03:37:11.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:11 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:11.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:11 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:11.870 INFO:tasks.workunit.client.0.smithi016.stdout: 1 52080 5.69 MB/sec warmup 52 sec latency 3706.267 ms 2024-04-13T03:37:11.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:11 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:12.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:12 smithi119 ceph-mon[35835]: pgmap v76: 129 pgs: 129 active+clean; 45 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-13T03:37:12.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:12 smithi016 ceph-mon[31772]: pgmap v76: 129 pgs: 129 active+clean; 45 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-13T03:37:12.870 INFO:tasks.workunit.client.0.smithi016.stdout: 1 52080 5.58 MB/sec warmup 53 sec latency 4706.393 ms 2024-04-13T03:37:12.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:12 smithi188 ceph-mon[36398]: pgmap v76: 129 pgs: 129 active+clean; 45 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-13T03:37:13.871 INFO:tasks.workunit.client.0.smithi016.stdout: 1 53053 5.61 MB/sec warmup 54 sec latency 4950.913 ms 2024-04-13T03:37:14.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:14 smithi119 ceph-mon[35835]: pgmap v77: 129 pgs: 129 active+clean; 43 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-13T03:37:14.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:14 smithi016 ceph-mon[31772]: pgmap v77: 129 pgs: 129 active+clean; 43 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-13T03:37:14.871 INFO:tasks.workunit.client.0.smithi016.stdout: 1 53835 5.60 MB/sec warmup 55 sec latency 16.815 ms 2024-04-13T03:37:14.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:14 smithi188 ceph-mon[36398]: pgmap v77: 129 pgs: 129 active+clean; 43 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-13T03:37:15.871 INFO:tasks.workunit.client.0.smithi016.stdout: 1 54027 5.50 MB/sec warmup 56 sec latency 779.126 ms 2024-04-13T03:37:16.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:16 smithi119 ceph-mon[35835]: pgmap v78: 129 pgs: 129 active+clean; 43 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-13T03:37:16.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:16 smithi016 ceph-mon[31772]: pgmap v78: 129 pgs: 129 active+clean; 43 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-13T03:37:16.871 INFO:tasks.workunit.client.0.smithi016.stdout: 1 54149 5.41 MB/sec warmup 57 sec latency 1659.584 ms 2024-04-13T03:37:16.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:16 smithi188 ceph-mon[36398]: pgmap v78: 129 pgs: 129 active+clean; 43 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.1 MiB/s wr, 122 op/s 2024-04-13T03:37:17.872 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55572 5.42 MB/sec warmup 58 sec latency 281.249 ms 2024-04-13T03:37:18.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:18 smithi119 ceph-mon[35835]: pgmap v79: 129 pgs: 129 active+clean; 43 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-04-13T03:37:18.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:18 smithi016 ceph-mon[31772]: pgmap v79: 129 pgs: 129 active+clean; 43 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-04-13T03:37:18.872 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55666 5.34 MB/sec warmup 59 sec latency 622.863 ms 2024-04-13T03:37:18.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:18 smithi188 ceph-mon[36398]: pgmap v79: 129 pgs: 129 active+clean; 43 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 1.8 MiB/s wr, 176 op/s 2024-04-13T03:37:19.872 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55666 5.25 MB/sec warmup 60 sec latency 1622.999 ms 2024-04-13T03:37:20.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:20 smithi016 ceph-mon[31772]: pgmap v80: 129 pgs: 129 active+clean; 43 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 831 KiB/s wr, 85 op/s 2024-04-13T03:37:20.872 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55666 5.16 MB/sec warmup 61 sec latency 2623.137 ms 2024-04-13T03:37:20.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:20 smithi188 ceph-mon[36398]: pgmap v80: 129 pgs: 129 active+clean; 43 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 831 KiB/s wr, 85 op/s 2024-04-13T03:37:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:20 smithi119 ceph-mon[35835]: pgmap v80: 129 pgs: 129 active+clean; 43 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 831 KiB/s wr, 85 op/s 2024-04-13T03:37:21.872 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55666 5.08 MB/sec warmup 62 sec latency 3623.260 ms 2024-04-13T03:37:22.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:22 smithi016 ceph-mon[31772]: pgmap v81: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-13T03:37:22.872 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55666 5.00 MB/sec warmup 63 sec latency 4623.413 ms 2024-04-13T03:37:22.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:22 smithi188 ceph-mon[36398]: pgmap v81: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-13T03:37:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:22 smithi119 ceph-mon[35835]: pgmap v81: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-13T03:37:23.872 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55727 4.93 MB/sec warmup 64 sec latency 4863.254 ms 2024-04-13T03:37:24.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:24 smithi016 ceph-mon[31772]: pgmap v82: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 999 KiB/s wr, 82 op/s 2024-04-13T03:37:24.872 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55727 4.85 MB/sec warmup 65 sec latency 1664.678 ms 2024-04-13T03:37:24.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:24 smithi188 ceph-mon[36398]: pgmap v82: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 999 KiB/s wr, 82 op/s 2024-04-13T03:37:25.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:24 smithi119 ceph-mon[35835]: pgmap v82: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 999 KiB/s wr, 82 op/s 2024-04-13T03:37:25.873 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55727 4.78 MB/sec warmup 66 sec latency 2664.806 ms 2024-04-13T03:37:26.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:26 smithi016 ceph-mon[31772]: pgmap v83: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 988 KiB/s wr, 81 op/s 2024-04-13T03:37:26.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:26 smithi188 ceph-mon[36398]: pgmap v83: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 988 KiB/s wr, 81 op/s 2024-04-13T03:37:26.873 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55727 4.71 MB/sec warmup 67 sec latency 3664.972 ms 2024-04-13T03:37:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:26 smithi119 ceph-mon[35835]: pgmap v83: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 988 KiB/s wr, 81 op/s 2024-04-13T03:37:27.873 INFO:tasks.workunit.client.0.smithi016.stdout: 1 55727 4.64 MB/sec warmup 68 sec latency 4665.110 ms 2024-04-13T03:37:28.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:28 smithi016 ceph-mon[31772]: pgmap v84: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-04-13T03:37:28.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:28 smithi188 ceph-mon[36398]: pgmap v84: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-04-13T03:37:28.873 INFO:tasks.workunit.client.0.smithi016.stdout: 1 56865 4.72 MB/sec warmup 69 sec latency 4904.251 ms 2024-04-13T03:37:29.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:28 smithi119 ceph-mon[35835]: pgmap v84: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-04-13T03:37:29.873 INFO:tasks.workunit.client.0.smithi016.stdout: 1 57503 4.67 MB/sec warmup 70 sec latency 108.514 ms 2024-04-13T03:37:30.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:30 smithi016 ceph-mon[31772]: pgmap v85: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 364 KiB/s wr, 38 op/s 2024-04-13T03:37:30.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:30 smithi188 ceph-mon[36398]: pgmap v85: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 364 KiB/s wr, 38 op/s 2024-04-13T03:37:30.873 INFO:tasks.workunit.client.0.smithi016.stdout: 1 57503 4.61 MB/sec warmup 71 sec latency 1108.665 ms 2024-04-13T03:37:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:30 smithi119 ceph-mon[35835]: pgmap v85: 129 pgs: 129 active+clean; 44 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 364 KiB/s wr, 38 op/s 2024-04-13T03:37:31.874 INFO:tasks.workunit.client.0.smithi016.stdout: 1 57605 4.55 MB/sec warmup 72 sec latency 1987.376 ms 2024-04-13T03:37:32.872 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:32 smithi119 ceph-mon[35835]: pgmap v86: 129 pgs: 129 active+clean; 45 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 466 KiB/s wr, 67 op/s 2024-04-13T03:37:32.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:32 smithi188 ceph-mon[36398]: pgmap v86: 129 pgs: 129 active+clean; 45 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 466 KiB/s wr, 67 op/s 2024-04-13T03:37:32.874 INFO:tasks.workunit.client.0.smithi016.stdout: 1 59209 4.58 MB/sec warmup 73 sec latency 80.148 ms 2024-04-13T03:37:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:32 smithi016 ceph-mon[31772]: pgmap v86: 129 pgs: 129 active+clean; 45 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 466 KiB/s wr, 67 op/s 2024-04-13T03:37:33.874 INFO:tasks.workunit.client.0.smithi016.stdout: 1 60548 4.67 MB/sec warmup 74 sec latency 15.013 ms 2024-04-13T03:37:34.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:34 smithi188 ceph-mon[36398]: pgmap v87: 129 pgs: 129 active+clean; 45 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 154 KiB/s wr, 40 op/s 2024-04-13T03:37:34.874 INFO:tasks.workunit.client.0.smithi016.stdout: 1 61051 4.62 MB/sec warmup 75 sec latency 171.489 ms 2024-04-13T03:37:35.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:34 smithi119 ceph-mon[35835]: pgmap v87: 129 pgs: 129 active+clean; 45 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 154 KiB/s wr, 40 op/s 2024-04-13T03:37:35.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:34 smithi016 ceph-mon[31772]: pgmap v87: 129 pgs: 129 active+clean; 45 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 154 KiB/s wr, 40 op/s 2024-04-13T03:37:35.874 INFO:tasks.workunit.client.0.smithi016.stdout: 1 61051 4.56 MB/sec warmup 76 sec latency 1171.652 ms 2024-04-13T03:37:36.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:36 smithi188 ceph-mon[36398]: pgmap v88: 129 pgs: 129 active+clean; 45 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 154 KiB/s wr, 39 op/s 2024-04-13T03:37:36.874 INFO:tasks.workunit.client.0.smithi016.stdout: 1 61119 4.50 MB/sec warmup 77 sec latency 2052.205 ms 2024-04-13T03:37:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:36 smithi119 ceph-mon[35835]: pgmap v88: 129 pgs: 129 active+clean; 45 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 154 KiB/s wr, 39 op/s 2024-04-13T03:37:37.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:36 smithi016 ceph-mon[31772]: pgmap v88: 129 pgs: 129 active+clean; 45 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 154 KiB/s wr, 39 op/s 2024-04-13T03:37:37.875 INFO:tasks.workunit.client.0.smithi016.stdout: 1 61119 4.44 MB/sec warmup 78 sec latency 1027.431 ms 2024-04-13T03:37:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:38 smithi188 ceph-mon[36398]: pgmap v89: 129 pgs: 129 active+clean; 47 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 706 KiB/s wr, 97 op/s 2024-04-13T03:37:38.874 INFO:tasks.workunit.client.0.smithi016.stdout: 1 61119 4.38 MB/sec warmup 79 sec latency 2027.541 ms 2024-04-13T03:37:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:38 smithi119 ceph-mon[35835]: pgmap v89: 129 pgs: 129 active+clean; 47 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 706 KiB/s wr, 97 op/s 2024-04-13T03:37:39.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:38 smithi016 ceph-mon[31772]: pgmap v89: 129 pgs: 129 active+clean; 47 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 706 KiB/s wr, 97 op/s 2024-04-13T03:37:39.875 INFO:tasks.workunit.client.0.smithi016.stdout: 1 61119 4.33 MB/sec warmup 80 sec latency 3027.670 ms 2024-04-13T03:37:40.875 INFO:tasks.workunit.client.0.smithi016.stdout: 1 61119 4.28 MB/sec warmup 81 sec latency 4027.783 ms 2024-04-13T03:37:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:40 smithi119 ceph-mon[35835]: pgmap v90: 129 pgs: 129 active+clean; 47 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 654 KiB/s wr, 86 op/s 2024-04-13T03:37:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:40 smithi016 ceph-mon[31772]: pgmap v90: 129 pgs: 129 active+clean; 47 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 654 KiB/s wr, 86 op/s 2024-04-13T03:37:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:40 smithi188 ceph-mon[36398]: pgmap v90: 129 pgs: 129 active+clean; 47 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 654 KiB/s wr, 86 op/s 2024-04-13T03:37:41.875 INFO:tasks.workunit.client.0.smithi016.stdout: 1 61243 4.23 MB/sec warmup 82 sec latency 4904.055 ms 2024-04-13T03:37:42.875 INFO:tasks.workunit.client.0.smithi016.stdout: 1 62827 4.26 MB/sec warmup 83 sec latency 23.044 ms 2024-04-13T03:37:42.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:42 smithi016 ceph-mon[31772]: pgmap v91: 129 pgs: 129 active+clean; 48 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 924 KiB/s wr, 105 op/s 2024-04-13T03:37:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:42 smithi119 ceph-mon[35835]: pgmap v91: 129 pgs: 129 active+clean; 48 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 924 KiB/s wr, 105 op/s 2024-04-13T03:37:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:42 smithi188 ceph-mon[36398]: pgmap v91: 129 pgs: 129 active+clean; 48 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 924 KiB/s wr, 105 op/s 2024-04-13T03:37:43.875 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64165 4.34 MB/sec warmup 84 sec latency 15.159 ms 2024-04-13T03:37:44.875 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64536 4.30 MB/sec warmup 85 sec latency 394.769 ms 2024-04-13T03:37:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:44 smithi119 ceph-mon[35835]: pgmap v92: 129 pgs: 129 active+clean; 48 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 825 KiB/s wr, 77 op/s 2024-04-13T03:37:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:44 smithi016 ceph-mon[31772]: pgmap v92: 129 pgs: 129 active+clean; 48 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 825 KiB/s wr, 77 op/s 2024-04-13T03:37:45.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:44 smithi188 ceph-mon[36398]: pgmap v92: 129 pgs: 129 active+clean; 48 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 825 KiB/s wr, 77 op/s 2024-04-13T03:37:45.876 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64536 4.25 MB/sec warmup 86 sec latency 1394.934 ms 2024-04-13T03:37:46.876 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64592 4.20 MB/sec warmup 87 sec latency 2275.452 ms 2024-04-13T03:37:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:46 smithi119 ceph-mon[35835]: pgmap v93: 129 pgs: 129 active+clean; 48 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 825 KiB/s wr, 77 op/s 2024-04-13T03:37:47.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:46 smithi016 ceph-mon[31772]: pgmap v93: 129 pgs: 129 active+clean; 48 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 825 KiB/s wr, 77 op/s 2024-04-13T03:37:47.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:46 smithi188 ceph-mon[36398]: pgmap v93: 129 pgs: 129 active+clean; 48 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 825 KiB/s wr, 77 op/s 2024-04-13T03:37:47.876 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64592 4.15 MB/sec warmup 88 sec latency 1050.280 ms 2024-04-13T03:37:48.876 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64592 4.11 MB/sec warmup 89 sec latency 2050.420 ms 2024-04-13T03:37:49.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:48 smithi119 ceph-mon[35835]: pgmap v94: 129 pgs: 129 active+clean; 49 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 1.0 MiB/s wr, 123 op/s 2024-04-13T03:37:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:48 smithi016 ceph-mon[31772]: pgmap v94: 129 pgs: 129 active+clean; 49 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 1.0 MiB/s wr, 123 op/s 2024-04-13T03:37:49.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:48 smithi188 ceph-mon[36398]: pgmap v94: 129 pgs: 129 active+clean; 49 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 1.0 MiB/s wr, 123 op/s 2024-04-13T03:37:49.876 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64592 4.06 MB/sec warmup 90 sec latency 3050.517 ms 2024-04-13T03:37:50.876 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64592 4.02 MB/sec warmup 91 sec latency 4050.663 ms 2024-04-13T03:37:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:50 smithi119 ceph-mon[35835]: pgmap v95: 129 pgs: 129 active+clean; 49 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 479 KiB/s wr, 65 op/s 2024-04-13T03:37:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:50 smithi016 ceph-mon[31772]: pgmap v95: 129 pgs: 129 active+clean; 49 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 479 KiB/s wr, 65 op/s 2024-04-13T03:37:51.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:50 smithi188 ceph-mon[36398]: pgmap v95: 129 pgs: 129 active+clean; 49 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 479 KiB/s wr, 65 op/s 2024-04-13T03:37:51.876 INFO:tasks.workunit.client.0.smithi016.stdout: 1 64682 3.97 MB/sec warmup 92 sec latency 4925.113 ms 2024-04-13T03:37:52.877 INFO:tasks.workunit.client.0.smithi016.stdout: 1 66306 4.00 MB/sec warmup 93 sec latency 25.651 ms 2024-04-13T03:37:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:52 smithi119 ceph-mon[35835]: pgmap v96: 129 pgs: 129 active+clean; 49 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 742 KiB/s wr, 81 op/s 2024-04-13T03:37:53.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:52 smithi016 ceph-mon[31772]: pgmap v96: 129 pgs: 129 active+clean; 49 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 742 KiB/s wr, 81 op/s 2024-04-13T03:37:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:52 smithi188 ceph-mon[36398]: pgmap v96: 129 pgs: 129 active+clean; 49 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 742 KiB/s wr, 81 op/s 2024-04-13T03:37:53.877 INFO:tasks.workunit.client.0.smithi016.stdout: 1 67634 4.08 MB/sec warmup 94 sec latency 16.701 ms 2024-04-13T03:37:54.023 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:53 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:54.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:53 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:54.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:53 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:54.877 INFO:tasks.workunit.client.0.smithi016.stdout: 1 68210 4.05 MB/sec warmup 95 sec latency 15.950 ms 2024-04-13T03:37:55.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:54 smithi119 ceph-mon[35835]: pgmap v97: 129 pgs: 129 active+clean; 49 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 473 KiB/s wr, 63 op/s 2024-04-13T03:37:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:54 smithi016 ceph-mon[31772]: pgmap v97: 129 pgs: 129 active+clean; 49 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 473 KiB/s wr, 63 op/s 2024-04-13T03:37:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:54 smithi188 ceph-mon[36398]: pgmap v97: 129 pgs: 129 active+clean; 49 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 473 KiB/s wr, 63 op/s 2024-04-13T03:37:55.878 INFO:tasks.workunit.client.0.smithi016.stdout: 1 68333 4.01 MB/sec warmup 96 sec latency 879.422 ms 2024-04-13T03:37:56.877 INFO:tasks.workunit.client.0.smithi016.stdout: 1 68333 3.97 MB/sec warmup 97 sec latency 1879.522 ms 2024-04-13T03:37:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:56 smithi119 ceph-mon[35835]: pgmap v98: 129 pgs: 129 active+clean; 49 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 470 KiB/s wr, 63 op/s 2024-04-13T03:37:57.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:56 smithi016 ceph-mon[31772]: pgmap v98: 129 pgs: 129 active+clean; 49 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 470 KiB/s wr, 63 op/s 2024-04-13T03:37:57.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:56 smithi188 ceph-mon[36398]: pgmap v98: 129 pgs: 129 active+clean; 49 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 470 KiB/s wr, 63 op/s 2024-04-13T03:37:57.877 INFO:tasks.workunit.client.0.smithi016.stdout: 1 68333 3.93 MB/sec warmup 98 sec latency 2879.640 ms 2024-04-13T03:37:58.877 INFO:tasks.workunit.client.0.smithi016.stdout: 1 68333 3.89 MB/sec warmup 99 sec latency 3879.778 ms 2024-04-13T03:37:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:58 smithi119 ceph-mon[35835]: pgmap v99: 129 pgs: 129 active+clean; 51 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 732 KiB/s wr, 113 op/s 2024-04-13T03:37:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:58 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:58 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:37:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:58 smithi016 ceph-mon[31772]: pgmap v99: 129 pgs: 129 active+clean; 51 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 732 KiB/s wr, 113 op/s 2024-04-13T03:37:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:58 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:59.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:58 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:37:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:58 smithi188 ceph-mon[36398]: pgmap v99: 129 pgs: 129 active+clean; 51 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 732 KiB/s wr, 113 op/s 2024-04-13T03:37:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:58 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:37:59.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:58 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:37:59.877 INFO:tasks.workunit.client.0.smithi016.stdout: 1 68399 3.85 MB/sec warmup 100 sec latency 3917.611 ms 2024-04-13T03:38:00.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:59 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:38:00.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:59 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:38:00.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:37:59 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:38:00.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:59 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:38:00.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:59 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:38:00.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:37:59 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:38:00.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:59 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:38:00.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:59 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:38:00.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:37:59 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:38:00.878 INFO:tasks.workunit.client.0.smithi016.stdout: 1 68429 3.81 MB/sec warmup 101 sec latency 1765.276 ms 2024-04-13T03:38:01.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:00 smithi119 ceph-mon[35835]: pgmap v100: 129 pgs: 129 active+clean; 51 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 525 KiB/s wr, 67 op/s 2024-04-13T03:38:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:00 smithi016 ceph-mon[31772]: pgmap v100: 129 pgs: 129 active+clean; 51 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 525 KiB/s wr, 67 op/s 2024-04-13T03:38:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:00 smithi188 ceph-mon[36398]: pgmap v100: 129 pgs: 129 active+clean; 51 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 525 KiB/s wr, 67 op/s 2024-04-13T03:38:01.878 INFO:tasks.workunit.client.0.smithi016.stdout: 1 68575 3.79 MB/sec warmup 102 sec latency 85.720 ms 2024-04-13T03:38:02.878 INFO:tasks.workunit.client.0.smithi016.stdout: 1 69990 3.81 MB/sec warmup 103 sec latency 49.024 ms 2024-04-13T03:38:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:02 smithi119 ceph-mon[35835]: pgmap v101: 129 pgs: 129 active+clean; 51 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 660 KiB/s wr, 104 op/s 2024-04-13T03:38:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:02 smithi016 ceph-mon[31772]: pgmap v101: 129 pgs: 129 active+clean; 51 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 660 KiB/s wr, 104 op/s 2024-04-13T03:38:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:02 smithi188 ceph-mon[36398]: pgmap v101: 129 pgs: 129 active+clean; 51 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 660 KiB/s wr, 104 op/s 2024-04-13T03:38:03.878 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71206 3.88 MB/sec warmup 104 sec latency 21.039 ms 2024-04-13T03:38:04.878 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71629 3.84 MB/sec warmup 105 sec latency 238.137 ms 2024-04-13T03:38:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:04 smithi119 ceph-mon[35835]: pgmap v102: 129 pgs: 129 active+clean; 51 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 405 KiB/s wr, 89 op/s 2024-04-13T03:38:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:04 smithi016 ceph-mon[31772]: pgmap v102: 129 pgs: 129 active+clean; 51 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 405 KiB/s wr, 89 op/s 2024-04-13T03:38:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:04 smithi188 ceph-mon[36398]: pgmap v102: 129 pgs: 129 active+clean; 51 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 405 KiB/s wr, 89 op/s 2024-04-13T03:38:05.879 INFO:tasks.workunit.client.0.smithi016.stdout: 1 71629 3.81 MB/sec warmup 106 sec latency 1238.263 ms 2024-04-13T03:38:06.878 INFO:tasks.workunit.client.0.smithi016.stdout: 1 72194 3.79 MB/sec warmup 107 sec latency 1580.921 ms 2024-04-13T03:38:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:06 smithi119 ceph-mon[35835]: pgmap v103: 129 pgs: 129 active+clean; 51 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 404 KiB/s wr, 89 op/s 2024-04-13T03:38:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:06 smithi016 ceph-mon[31772]: pgmap v103: 129 pgs: 129 active+clean; 51 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 404 KiB/s wr, 89 op/s 2024-04-13T03:38:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:06 smithi188 ceph-mon[36398]: pgmap v103: 129 pgs: 129 active+clean; 51 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 404 KiB/s wr, 89 op/s 2024-04-13T03:38:07.878 INFO:tasks.workunit.client.0.smithi016.stdout: 1 73872 3.84 MB/sec warmup 108 sec latency 47.004 ms 2024-04-13T03:38:08.879 INFO:tasks.workunit.client.0.smithi016.stdout: 1 74869 3.87 MB/sec warmup 109 sec latency 29.006 ms 2024-04-13T03:38:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:08 smithi119 ceph-mon[35835]: pgmap v104: 129 pgs: 129 active+clean; 54 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 930 KiB/s wr, 147 op/s 2024-04-13T03:38:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:08 smithi016 ceph-mon[31772]: pgmap v104: 129 pgs: 129 active+clean; 54 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 930 KiB/s wr, 147 op/s 2024-04-13T03:38:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:08 smithi188 ceph-mon[36398]: pgmap v104: 129 pgs: 129 active+clean; 54 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 930 KiB/s wr, 147 op/s 2024-04-13T03:38:09.879 INFO:tasks.workunit.client.0.smithi016.stdout: 1 75533 3.85 MB/sec warmup 110 sec latency 19.589 ms 2024-04-13T03:38:10.879 INFO:tasks.workunit.client.0.smithi016.stdout: 1 77101 3.88 MB/sec warmup 111 sec latency 16.081 ms 2024-04-13T03:38:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:10 smithi119 ceph-mon[35835]: pgmap v105: 129 pgs: 129 active+clean; 54 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 668 KiB/s wr, 96 op/s 2024-04-13T03:38:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:10 smithi016 ceph-mon[31772]: pgmap v105: 129 pgs: 129 active+clean; 54 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 668 KiB/s wr, 96 op/s 2024-04-13T03:38:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:10 smithi188 ceph-mon[36398]: pgmap v105: 129 pgs: 129 active+clean; 54 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 668 KiB/s wr, 96 op/s 2024-04-13T03:38:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: 1 78327 3.94 MB/sec warmup 112 sec latency 15.295 ms 2024-04-13T03:38:12.879 INFO:tasks.workunit.client.0.smithi016.stdout: 1 78867 3.91 MB/sec warmup 113 sec latency 25.186 ms 2024-04-13T03:38:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:12 smithi119 ceph-mon[35835]: pgmap v106: 129 pgs: 129 active+clean; 57 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.2 MiB/s wr, 164 op/s 2024-04-13T03:38:13.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:12 smithi016 ceph-mon[31772]: pgmap v106: 129 pgs: 129 active+clean; 57 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.2 MiB/s wr, 164 op/s 2024-04-13T03:38:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:12 smithi188 ceph-mon[36398]: pgmap v106: 129 pgs: 129 active+clean; 57 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 1.2 MiB/s wr, 164 op/s 2024-04-13T03:38:13.879 INFO:tasks.workunit.client.0.smithi016.stdout: 1 80447 3.94 MB/sec warmup 114 sec latency 39.311 ms 2024-04-13T03:38:14.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 81631 3.99 MB/sec warmup 115 sec latency 21.467 ms 2024-04-13T03:38:15.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:14 smithi119 ceph-mon[35835]: pgmap v107: 129 pgs: 129 active+clean; 57 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-13T03:38:15.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:14 smithi016 ceph-mon[31772]: pgmap v107: 129 pgs: 129 active+clean; 57 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-13T03:38:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:14 smithi188 ceph-mon[36398]: pgmap v107: 129 pgs: 129 active+clean; 57 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-13T03:38:15.880 INFO:tasks.workunit.client.0.smithi016.stdout: 1 82232 3.97 MB/sec warmup 116 sec latency 15.054 ms 2024-04-13T03:38:16.880 INFO:tasks.workunit.client.0.smithi016.stdout: 1 82302 3.94 MB/sec warmup 117 sec latency 949.510 ms 2024-04-13T03:38:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:16 smithi119 ceph-mon[35835]: pgmap v108: 129 pgs: 129 active+clean; 57 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-13T03:38:17.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:16 smithi016 ceph-mon[31772]: pgmap v108: 129 pgs: 129 active+clean; 57 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-13T03:38:17.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:16 smithi188 ceph-mon[36398]: pgmap v108: 129 pgs: 129 active+clean; 57 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-13T03:38:17.880 INFO:tasks.workunit.client.0.smithi016.stdout: 1 82302 3.91 MB/sec warmup 118 sec latency 1949.659 ms 2024-04-13T03:38:18.880 INFO:tasks.workunit.client.0.smithi016.stdout: 1 82302 3.87 MB/sec warmup 119 sec latency 2949.811 ms 2024-04-13T03:38:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:18 smithi119 ceph-mon[35835]: pgmap v109: 129 pgs: 129 active+clean; 59 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 189 op/s 2024-04-13T03:38:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:18 smithi016 ceph-mon[31772]: pgmap v109: 129 pgs: 129 active+clean; 59 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 189 op/s 2024-04-13T03:38:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:18 smithi188 ceph-mon[36398]: pgmap v109: 129 pgs: 129 active+clean; 59 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 189 op/s 2024-04-13T03:38:20.880 INFO:tasks.workunit.client.0.smithi016.stdout: 1 85132 12.01 MB/sec execute 1 sec latency 2985.250 ms 2024-04-13T03:38:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:20 smithi119 ceph-mon[35835]: pgmap v110: 129 pgs: 129 active+clean; 59 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 962 KiB/s wr, 131 op/s 2024-04-13T03:38:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:20 smithi016 ceph-mon[31772]: pgmap v110: 129 pgs: 129 active+clean; 59 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 962 KiB/s wr, 131 op/s 2024-04-13T03:38:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:20 smithi188 ceph-mon[36398]: pgmap v110: 129 pgs: 129 active+clean; 59 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 962 KiB/s wr, 131 op/s 2024-04-13T03:38:21.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 85898 6.99 MB/sec execute 2 sec latency 14.323 ms 2024-04-13T03:38:22.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 85967 4.71 MB/sec execute 3 sec latency 887.984 ms 2024-04-13T03:38:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:22 smithi119 ceph-mon[35835]: pgmap v111: 129 pgs: 129 active+clean; 61 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 1.3 MiB/s wr, 176 op/s 2024-04-13T03:38:23.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:22 smithi016 ceph-mon[31772]: pgmap v111: 129 pgs: 129 active+clean; 61 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 1.3 MiB/s wr, 176 op/s 2024-04-13T03:38:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:22 smithi188 ceph-mon[36398]: pgmap v111: 129 pgs: 129 active+clean; 61 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 1.3 MiB/s wr, 176 op/s 2024-04-13T03:38:23.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 85967 3.54 MB/sec execute 4 sec latency 1888.146 ms 2024-04-13T03:38:24.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:23 smithi119 ceph-mon[35835]: pgmap v112: 129 pgs: 129 active+clean; 61 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 762 KiB/s wr, 109 op/s 2024-04-13T03:38:24.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:23 smithi016 ceph-mon[31772]: pgmap v112: 129 pgs: 129 active+clean; 61 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 762 KiB/s wr, 109 op/s 2024-04-13T03:38:24.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:23 smithi188 ceph-mon[36398]: pgmap v112: 129 pgs: 129 active+clean; 61 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 762 KiB/s wr, 109 op/s 2024-04-13T03:38:24.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 87606 4.18 MB/sec execute 5 sec latency 1917.557 ms 2024-04-13T03:38:25.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 88971 5.42 MB/sec execute 6 sec latency 14.475 ms 2024-04-13T03:38:26.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:26 smithi119 ceph-mon[35835]: pgmap v113: 129 pgs: 129 active+clean; 61 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 762 KiB/s wr, 109 op/s 2024-04-13T03:38:26.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:26 smithi016 ceph-mon[31772]: pgmap v113: 129 pgs: 129 active+clean; 61 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 762 KiB/s wr, 109 op/s 2024-04-13T03:38:26.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:26 smithi188 ceph-mon[36398]: pgmap v113: 129 pgs: 129 active+clean; 61 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 762 KiB/s wr, 109 op/s 2024-04-13T03:38:26.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 89392 4.71 MB/sec execute 7 sec latency 367.512 ms 2024-04-13T03:38:27.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 89392 4.12 MB/sec execute 8 sec latency 1367.673 ms 2024-04-13T03:38:28.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:28 smithi119 ceph-mon[35835]: pgmap v114: 129 pgs: 129 active+clean; 63 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.1 MiB/s wr, 164 op/s 2024-04-13T03:38:28.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:28 smithi016 ceph-mon[31772]: pgmap v114: 129 pgs: 129 active+clean; 63 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.1 MiB/s wr, 164 op/s 2024-04-13T03:38:28.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:28 smithi188 ceph-mon[36398]: pgmap v114: 129 pgs: 129 active+clean; 63 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.1 MiB/s wr, 164 op/s 2024-04-13T03:38:28.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 89392 3.66 MB/sec execute 9 sec latency 2367.839 ms 2024-04-13T03:38:29.882 INFO:tasks.workunit.client.0.smithi016.stdout: 1 91061 3.97 MB/sec execute 10 sec latency 2400.728 ms 2024-04-13T03:38:30.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:30 smithi016 ceph-mon[31772]: pgmap v115: 129 pgs: 129 active+clean; 63 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 726 KiB/s wr, 100 op/s 2024-04-13T03:38:30.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:30 smithi188 ceph-mon[36398]: pgmap v115: 129 pgs: 129 active+clean; 63 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 726 KiB/s wr, 100 op/s 2024-04-13T03:38:30.881 INFO:tasks.workunit.client.0.smithi016.stdout: 1 92471 4.70 MB/sec execute 11 sec latency 12.505 ms 2024-04-13T03:38:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:30 smithi119 ceph-mon[35835]: pgmap v115: 129 pgs: 129 active+clean; 63 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 726 KiB/s wr, 100 op/s 2024-04-13T03:38:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: 1 92936 4.34 MB/sec execute 12 sec latency 252.036 ms 2024-04-13T03:38:32.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:32 smithi016 ceph-mon[31772]: pgmap v116: 129 pgs: 129 active+clean; 65 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-13T03:38:32.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:32 smithi188 ceph-mon[36398]: pgmap v116: 129 pgs: 129 active+clean; 65 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-13T03:38:32.873 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:32 smithi119 ceph-mon[35835]: pgmap v116: 129 pgs: 129 active+clean; 65 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-13T03:38:32.882 INFO:tasks.workunit.client.0.smithi016.stdout: 1 92936 4.01 MB/sec execute 13 sec latency 1252.198 ms 2024-04-13T03:38:33.882 INFO:tasks.workunit.client.0.smithi016.stdout: 1 92936 3.72 MB/sec execute 14 sec latency 2252.343 ms 2024-04-13T03:38:34.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:34 smithi016 ceph-mon[31772]: pgmap v117: 129 pgs: 129 active+clean; 65 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 733 KiB/s wr, 103 op/s 2024-04-13T03:38:34.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:34 smithi188 ceph-mon[36398]: pgmap v117: 129 pgs: 129 active+clean; 65 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 733 KiB/s wr, 103 op/s 2024-04-13T03:38:34.882 INFO:tasks.workunit.client.0.smithi016.stdout: 1 94567 3.92 MB/sec execute 15 sec latency 2282.259 ms 2024-04-13T03:38:35.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:34 smithi119 ceph-mon[35835]: pgmap v117: 129 pgs: 129 active+clean; 65 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 733 KiB/s wr, 103 op/s 2024-04-13T03:38:35.882 INFO:tasks.workunit.client.0.smithi016.stdout: 1 95973 4.36 MB/sec execute 16 sec latency 13.205 ms 2024-04-13T03:38:36.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:36 smithi016 ceph-mon[31772]: pgmap v118: 129 pgs: 129 active+clean; 65 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 733 KiB/s wr, 103 op/s 2024-04-13T03:38:36.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:36 smithi188 ceph-mon[36398]: pgmap v118: 129 pgs: 129 active+clean; 65 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 733 KiB/s wr, 103 op/s 2024-04-13T03:38:36.882 INFO:tasks.workunit.client.0.smithi016.stdout: 1 96607 4.21 MB/sec execute 17 sec latency 74.074 ms 2024-04-13T03:38:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:36 smithi119 ceph-mon[35835]: pgmap v118: 129 pgs: 129 active+clean; 65 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 733 KiB/s wr, 103 op/s 2024-04-13T03:38:37.883 INFO:tasks.workunit.client.0.smithi016.stdout: 1 96607 3.97 MB/sec execute 18 sec latency 1074.165 ms 2024-04-13T03:38:38.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:38 smithi016 ceph-mon[31772]: pgmap v119: 129 pgs: 129 active+clean; 67 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 1.1 MiB/s wr, 157 op/s 2024-04-13T03:38:38.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:38 smithi188 ceph-mon[36398]: pgmap v119: 129 pgs: 129 active+clean; 67 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 1.1 MiB/s wr, 157 op/s 2024-04-13T03:38:38.883 INFO:tasks.workunit.client.0.smithi016.stdout: 1 96607 3.76 MB/sec execute 19 sec latency 2074.354 ms 2024-04-13T03:38:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:38 smithi119 ceph-mon[35835]: pgmap v119: 129 pgs: 129 active+clean; 67 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 1.1 MiB/s wr, 157 op/s 2024-04-13T03:38:39.883 INFO:tasks.workunit.client.0.smithi016.stdout: 1 98272 3.91 MB/sec execute 20 sec latency 2099.973 ms 2024-04-13T03:38:40.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:40 smithi016 ceph-mon[31772]: pgmap v120: 129 pgs: 129 active+clean; 67 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 844 KiB/s wr, 102 op/s 2024-04-13T03:38:40.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:40 smithi188 ceph-mon[36398]: pgmap v120: 129 pgs: 129 active+clean; 67 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 844 KiB/s wr, 102 op/s 2024-04-13T03:38:40.883 INFO:tasks.workunit.client.0.smithi016.stdout: 1 99657 4.28 MB/sec execute 21 sec latency 18.510 ms 2024-04-13T03:38:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:40 smithi119 ceph-mon[35835]: pgmap v120: 129 pgs: 129 active+clean; 67 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 844 KiB/s wr, 102 op/s 2024-04-13T03:38:41.883 INFO:tasks.workunit.client.0.smithi016.stdout: 1 100469 4.19 MB/sec execute 22 sec latency 15.169 ms 2024-04-13T03:38:42.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:42 smithi188 ceph-mon[36398]: pgmap v121: 129 pgs: 129 active+clean; 69 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-13T03:38:42.883 INFO:tasks.workunit.client.0.smithi016.stdout: 1 102291 4.47 MB/sec execute 23 sec latency 16.789 ms 2024-04-13T03:38:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:42 smithi119 ceph-mon[35835]: pgmap v121: 129 pgs: 129 active+clean; 69 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-13T03:38:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:42 smithi016 ceph-mon[31772]: pgmap v121: 129 pgs: 129 active+clean; 69 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-04-13T03:38:43.883 INFO:tasks.workunit.client.0.smithi016.stdout: 1 103343 4.56 MB/sec execute 24 sec latency 24.594 ms 2024-04-13T03:38:44.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:44 smithi188 ceph-mon[36398]: pgmap v122: 129 pgs: 129 active+clean; 69 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:38:44.884 INFO:tasks.workunit.client.0.smithi016.stdout: 1 104469 4.58 MB/sec execute 25 sec latency 25.966 ms 2024-04-13T03:38:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:44 smithi119 ceph-mon[35835]: pgmap v122: 129 pgs: 129 active+clean; 69 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:38:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:44 smithi016 ceph-mon[31772]: pgmap v122: 129 pgs: 129 active+clean; 69 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:38:45.884 INFO:tasks.workunit.client.0.smithi016.stdout: 1 106278 4.76 MB/sec execute 26 sec latency 13.136 ms 2024-04-13T03:38:46.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:46 smithi188 ceph-mon[36398]: pgmap v123: 129 pgs: 129 active+clean; 69 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:38:46.884 INFO:tasks.workunit.client.0.smithi016.stdout: 1 107117 4.76 MB/sec execute 27 sec latency 17.366 ms 2024-04-13T03:38:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:46 smithi119 ceph-mon[35835]: pgmap v123: 129 pgs: 129 active+clean; 69 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:38:47.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:46 smithi016 ceph-mon[31772]: pgmap v123: 129 pgs: 129 active+clean; 69 MiB data, 617 MiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:38:47.884 INFO:tasks.workunit.client.0.smithi016.stdout: 1 108808 4.84 MB/sec execute 28 sec latency 14.729 ms 2024-04-13T03:38:48.884 INFO:tasks.workunit.client.0.smithi016.stdout: 1 110115 5.04 MB/sec execute 29 sec latency 49.480 ms 2024-04-13T03:38:49.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:48 smithi119 ceph-mon[35835]: pgmap v124: 129 pgs: 129 active+clean; 73 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 1.7 MiB/s wr, 202 op/s 2024-04-13T03:38:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:48 smithi016 ceph-mon[31772]: pgmap v124: 129 pgs: 129 active+clean; 73 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 1.7 MiB/s wr, 202 op/s 2024-04-13T03:38:49.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:48 smithi188 ceph-mon[36398]: pgmap v124: 129 pgs: 129 active+clean; 73 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 1.7 MiB/s wr, 202 op/s 2024-04-13T03:38:49.884 INFO:tasks.workunit.client.0.smithi016.stdout: 1 110803 4.94 MB/sec execute 30 sec latency 18.605 ms 2024-04-13T03:38:50.884 INFO:tasks.workunit.client.0.smithi016.stdout: 1 112467 5.01 MB/sec execute 31 sec latency 17.441 ms 2024-04-13T03:38:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:50 smithi119 ceph-mon[35835]: pgmap v125: 129 pgs: 129 active+clean; 73 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-04-13T03:38:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:50 smithi016 ceph-mon[31772]: pgmap v125: 129 pgs: 129 active+clean; 73 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-04-13T03:38:51.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:50 smithi188 ceph-mon[36398]: pgmap v125: 129 pgs: 129 active+clean; 73 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.3 MiB/s wr, 148 op/s 2024-04-13T03:38:51.884 INFO:tasks.workunit.client.0.smithi016.stdout: 1 113781 5.20 MB/sec execute 32 sec latency 15.640 ms 2024-04-13T03:38:52.885 INFO:tasks.workunit.client.0.smithi016.stdout: 1 114334 5.06 MB/sec execute 33 sec latency 179.469 ms 2024-04-13T03:38:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:52 smithi119 ceph-mon[35835]: pgmap v126: 129 pgs: 129 active+clean; 76 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 1.9 MiB/s wr, 224 op/s 2024-04-13T03:38:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:52 smithi016 ceph-mon[31772]: pgmap v126: 129 pgs: 129 active+clean; 76 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 1.9 MiB/s wr, 224 op/s 2024-04-13T03:38:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:52 smithi188 ceph-mon[36398]: pgmap v126: 129 pgs: 129 active+clean; 76 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 1.9 MiB/s wr, 224 op/s 2024-04-13T03:38:53.885 INFO:tasks.workunit.client.0.smithi016.stdout: 1 114334 4.92 MB/sec execute 34 sec latency 1179.640 ms 2024-04-13T03:38:54.885 INFO:tasks.workunit.client.0.smithi016.stdout: 1 116011 4.98 MB/sec execute 35 sec latency 1204.382 ms 2024-04-13T03:38:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:54 smithi188 ceph-mon[36398]: pgmap v127: 129 pgs: 129 active+clean; 76 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-04-13T03:38:55.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:54 smithi119 ceph-mon[35835]: pgmap v127: 129 pgs: 129 active+clean; 76 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-04-13T03:38:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:54 smithi016 ceph-mon[31772]: pgmap v127: 129 pgs: 129 active+clean; 76 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-04-13T03:38:55.885 INFO:tasks.workunit.client.0.smithi016.stdout: 1 117328 5.15 MB/sec execute 36 sec latency 21.157 ms 2024-04-13T03:38:56.885 INFO:tasks.workunit.client.0.smithi016.stdout: 1 118068 5.05 MB/sec execute 37 sec latency 13.816 ms 2024-04-13T03:38:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:56 smithi119 ceph-mon[35835]: pgmap v128: 129 pgs: 129 active+clean; 76 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-04-13T03:38:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:56 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:38:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:56 smithi016 ceph-mon[31772]: pgmap v128: 129 pgs: 129 active+clean; 76 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-04-13T03:38:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:56 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:38:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:56 smithi188 ceph-mon[36398]: pgmap v128: 129 pgs: 129 active+clean; 76 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 1.3 MiB/s wr, 165 op/s 2024-04-13T03:38:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:56 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:38:57.885 INFO:tasks.workunit.client.0.smithi016.stdout: 1 119865 5.18 MB/sec execute 38 sec latency 15.837 ms 2024-04-13T03:38:58.885 INFO:tasks.workunit.client.0.smithi016.stdout: 1 121004 5.25 MB/sec execute 39 sec latency 17.460 ms 2024-04-13T03:38:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:38:58 smithi119 ceph-mon[35835]: pgmap v129: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 233 op/s 2024-04-13T03:38:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:38:58 smithi016 ceph-mon[31772]: pgmap v129: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 233 op/s 2024-04-13T03:38:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:38:58 smithi188 ceph-mon[36398]: pgmap v129: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 233 op/s 2024-04-13T03:38:59.885 INFO:tasks.workunit.client.0.smithi016.stdout: 1 121424 5.13 MB/sec execute 40 sec latency 427.370 ms 2024-04-13T03:39:00.885 INFO:tasks.workunit.client.0.smithi016.stdout: 1 121424 5.01 MB/sec execute 41 sec latency 1427.477 ms 2024-04-13T03:39:01.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:00 smithi119 ceph-mon[35835]: pgmap v130: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-04-13T03:39:01.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:00 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:39:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:00 smithi016 ceph-mon[31772]: pgmap v130: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-04-13T03:39:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:00 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:39:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:00 smithi188 ceph-mon[36398]: pgmap v130: 129 pgs: 129 active+clean; 78 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.2 MiB/s wr, 144 op/s 2024-04-13T03:39:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:00 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:39:01.886 INFO:tasks.workunit.client.0.smithi016.stdout: 1 121665 4.92 MB/sec execute 42 sec latency 1473.387 ms 2024-04-13T03:39:02.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:01 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:39:02.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:01 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:39:02.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:01 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:39:02.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:01 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:39:02.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:01 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:39:02.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:01 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:39:02.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:01 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:39:02.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:01 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:39:02.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:01 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:39:02.886 INFO:tasks.workunit.client.0.smithi016.stdout: 1 123023 4.93 MB/sec execute 43 sec latency 86.925 ms 2024-04-13T03:39:02.972 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:02 smithi119 ceph-mon[35835]: pgmap v131: 129 pgs: 129 active+clean; 81 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-04-13T03:39:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:02 smithi016 ceph-mon[31772]: pgmap v131: 129 pgs: 129 active+clean; 81 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-04-13T03:39:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:02 smithi188 ceph-mon[36398]: pgmap v131: 129 pgs: 129 active+clean; 81 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-04-13T03:39:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124240 5.06 MB/sec execute 44 sec latency 57.488 ms 2024-04-13T03:39:04.886 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124851 4.98 MB/sec execute 45 sec latency 62.358 ms 2024-04-13T03:39:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:04 smithi119 ceph-mon[35835]: pgmap v132: 129 pgs: 129 active+clean; 81 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-13T03:39:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:04 smithi016 ceph-mon[31772]: pgmap v132: 129 pgs: 129 active+clean; 81 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-13T03:39:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:04 smithi188 ceph-mon[36398]: pgmap v132: 129 pgs: 129 active+clean; 81 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-13T03:39:05.886 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124905 4.88 MB/sec execute 46 sec latency 923.375 ms 2024-04-13T03:39:06.886 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124905 4.77 MB/sec execute 47 sec latency 1923.493 ms 2024-04-13T03:39:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:06 smithi119 ceph-mon[35835]: pgmap v133: 129 pgs: 129 active+clean; 81 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-13T03:39:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:06 smithi016 ceph-mon[31772]: pgmap v133: 129 pgs: 129 active+clean; 81 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-13T03:39:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:06 smithi188 ceph-mon[36398]: pgmap v133: 129 pgs: 129 active+clean; 81 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 1.1 MiB/s wr, 151 op/s 2024-04-13T03:39:07.886 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124905 4.67 MB/sec execute 48 sec latency 2923.626 ms 2024-04-13T03:39:08.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:07 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:08.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:07 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:08.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:07 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:08.886 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124905 4.58 MB/sec execute 49 sec latency 3923.795 ms 2024-04-13T03:39:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:08 smithi119 ceph-mon[35835]: pgmap v134: 129 pgs: 129 active+clean; 82 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 1.6 MiB/s wr, 217 op/s 2024-04-13T03:39:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:08 smithi016 ceph-mon[31772]: pgmap v134: 129 pgs: 129 active+clean; 82 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 1.6 MiB/s wr, 217 op/s 2024-04-13T03:39:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:08 smithi188 ceph-mon[36398]: pgmap v134: 129 pgs: 129 active+clean; 82 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 1.6 MiB/s wr, 217 op/s 2024-04-13T03:39:09.886 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124973 4.49 MB/sec execute 50 sec latency 3950.210 ms 2024-04-13T03:39:10.887 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124973 4.40 MB/sec execute 51 sec latency 1883.322 ms 2024-04-13T03:39:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:10 smithi119 ceph-mon[35835]: pgmap v135: 129 pgs: 129 active+clean; 82 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-13T03:39:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:10 smithi016 ceph-mon[31772]: pgmap v135: 129 pgs: 129 active+clean; 82 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-13T03:39:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:10 smithi188 ceph-mon[36398]: pgmap v135: 129 pgs: 129 active+clean; 82 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-13T03:39:11.887 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124973 4.32 MB/sec execute 52 sec latency 2883.512 ms 2024-04-13T03:39:12.006 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:11 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:12.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:11 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:12.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:11 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:12.887 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124973 4.24 MB/sec execute 53 sec latency 3883.622 ms 2024-04-13T03:39:13.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:12 smithi119 ceph-mon[35835]: pgmap v136: 129 pgs: 129 active+clean; 85 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.4 MiB/s wr, 167 op/s 2024-04-13T03:39:13.072 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:12 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:13.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:12 smithi016 ceph-mon[31772]: pgmap v136: 129 pgs: 129 active+clean; 85 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.4 MiB/s wr, 167 op/s 2024-04-13T03:39:13.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:12 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:13.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:12 smithi188 ceph-mon[36398]: pgmap v136: 129 pgs: 129 active+clean; 85 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.4 MiB/s wr, 167 op/s 2024-04-13T03:39:13.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:12 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:13.887 INFO:tasks.workunit.client.0.smithi016.stdout: 1 124973 4.16 MB/sec execute 54 sec latency 4883.758 ms 2024-04-13T03:39:14.887 INFO:tasks.workunit.client.0.smithi016.stdout: 1 126582 4.20 MB/sec execute 55 sec latency 4909.562 ms 2024-04-13T03:39:15.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:14 smithi119 ceph-mon[35835]: pgmap v137: 129 pgs: 129 active+clean; 85 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 864 KiB/s wr, 84 op/s 2024-04-13T03:39:15.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:14 smithi016 ceph-mon[31772]: pgmap v137: 129 pgs: 129 active+clean; 85 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 864 KiB/s wr, 84 op/s 2024-04-13T03:39:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:14 smithi188 ceph-mon[36398]: pgmap v137: 129 pgs: 129 active+clean; 85 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 864 KiB/s wr, 84 op/s 2024-04-13T03:39:15.887 INFO:tasks.workunit.client.0.smithi016.stdout: 1 127880 4.32 MB/sec execute 56 sec latency 27.754 ms 2024-04-13T03:39:16.887 INFO:tasks.workunit.client.0.smithi016.stdout: 1 128468 4.27 MB/sec execute 57 sec latency 16.649 ms 2024-04-13T03:39:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:16 smithi119 ceph-mon[35835]: pgmap v138: 129 pgs: 129 active+clean; 85 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 864 KiB/s wr, 84 op/s 2024-04-13T03:39:17.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:16 smithi016 ceph-mon[31772]: pgmap v138: 129 pgs: 129 active+clean; 85 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 864 KiB/s wr, 84 op/s 2024-04-13T03:39:17.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:16 smithi188 ceph-mon[36398]: pgmap v138: 129 pgs: 129 active+clean; 85 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 864 KiB/s wr, 84 op/s 2024-04-13T03:39:17.888 INFO:tasks.workunit.client.0.smithi016.stdout: 1 130082 4.31 MB/sec execute 58 sec latency 17.205 ms 2024-04-13T03:39:18.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:17 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:18.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:17 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:18.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:17 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:39:18.888 INFO:tasks.workunit.client.0.smithi016.stdout: 1 131226 4.42 MB/sec execute 59 sec latency 18.517 ms 2024-04-13T03:39:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:18 smithi119 ceph-mon[35835]: pgmap v139: 129 pgs: 129 active+clean; 85 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-04-13T03:39:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:18 smithi016 ceph-mon[31772]: pgmap v139: 129 pgs: 129 active+clean; 85 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-04-13T03:39:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:18 smithi188 ceph-mon[36398]: pgmap v139: 129 pgs: 129 active+clean; 85 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-04-13T03:39:19.888 INFO:tasks.workunit.client.0.smithi016.stdout: 1 131778 4.37 MB/sec execute 60 sec latency 56.632 ms 2024-04-13T03:39:20.888 INFO:tasks.workunit.client.0.smithi016.stdout: 1 132067 4.31 MB/sec execute 61 sec latency 673.138 ms 2024-04-13T03:39:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:20 smithi119 ceph-mon[35835]: pgmap v140: 129 pgs: 129 active+clean; 85 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 1.0 MiB/s wr, 77 op/s 2024-04-13T03:39:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:20 smithi016 ceph-mon[31772]: pgmap v140: 129 pgs: 129 active+clean; 85 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 1.0 MiB/s wr, 77 op/s 2024-04-13T03:39:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:20 smithi188 ceph-mon[36398]: pgmap v140: 129 pgs: 129 active+clean; 85 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 1.0 MiB/s wr, 77 op/s 2024-04-13T03:39:21.888 INFO:tasks.workunit.client.0.smithi016.stdout: 1 132067 4.24 MB/sec execute 62 sec latency 1673.291 ms 2024-04-13T03:39:22.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 132067 4.17 MB/sec execute 63 sec latency 2673.416 ms 2024-04-13T03:39:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:22 smithi119 ceph-mon[35835]: pgmap v141: 129 pgs: 129 active+clean; 87 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.8 MiB/s wr, 144 op/s 2024-04-13T03:39:23.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:22 smithi016 ceph-mon[31772]: pgmap v141: 129 pgs: 129 active+clean; 87 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.8 MiB/s wr, 144 op/s 2024-04-13T03:39:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:22 smithi188 ceph-mon[36398]: pgmap v141: 129 pgs: 129 active+clean; 87 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.8 MiB/s wr, 144 op/s 2024-04-13T03:39:23.888 INFO:tasks.workunit.client.0.smithi016.stdout: 1 132067 4.10 MB/sec execute 64 sec latency 3673.557 ms 2024-04-13T03:39:24.888 INFO:tasks.workunit.client.0.smithi016.stdout: 1 133655 4.14 MB/sec execute 65 sec latency 3698.892 ms 2024-04-13T03:39:25.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:24 smithi119 ceph-mon[35835]: pgmap v142: 129 pgs: 129 active+clean; 87 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-13T03:39:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:24 smithi016 ceph-mon[31772]: pgmap v142: 129 pgs: 129 active+clean; 87 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-13T03:39:25.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:24 smithi188 ceph-mon[36398]: pgmap v142: 129 pgs: 129 active+clean; 87 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-13T03:39:25.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 134851 4.24 MB/sec execute 66 sec latency 17.408 ms 2024-04-13T03:39:26.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 135422 4.20 MB/sec execute 67 sec latency 15.130 ms 2024-04-13T03:39:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:26 smithi119 ceph-mon[35835]: pgmap v143: 129 pgs: 129 active+clean; 87 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-13T03:39:27.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:26 smithi016 ceph-mon[31772]: pgmap v143: 129 pgs: 129 active+clean; 87 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-13T03:39:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:26 smithi188 ceph-mon[36398]: pgmap v143: 129 pgs: 129 active+clean; 87 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-13T03:39:27.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 136631 4.22 MB/sec execute 68 sec latency 16.610 ms 2024-04-13T03:39:28.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 138026 4.27 MB/sec execute 69 sec latency 20.430 ms 2024-04-13T03:39:29.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:28 smithi119 ceph-mon[35835]: pgmap v144: 129 pgs: 129 active+clean; 88 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-04-13T03:39:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:28 smithi016 ceph-mon[31772]: pgmap v144: 129 pgs: 129 active+clean; 88 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-04-13T03:39:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:28 smithi188 ceph-mon[36398]: pgmap v144: 129 pgs: 129 active+clean; 88 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.9 MiB/s wr, 175 op/s 2024-04-13T03:39:29.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 138808 4.29 MB/sec execute 70 sec latency 25.427 ms 2024-04-13T03:39:30.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 139803 4.30 MB/sec execute 71 sec latency 32.281 ms 2024-04-13T03:39:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:30 smithi119 ceph-mon[35835]: pgmap v145: 129 pgs: 129 active+clean; 88 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-04-13T03:39:31.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:30 smithi016 ceph-mon[31772]: pgmap v145: 129 pgs: 129 active+clean; 88 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-04-13T03:39:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:30 smithi188 ceph-mon[36398]: pgmap v145: 129 pgs: 129 active+clean; 88 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.2 MiB/s wr, 116 op/s 2024-04-13T03:39:31.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 141362 4.35 MB/sec execute 72 sec latency 29.140 ms 2024-04-13T03:39:32.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 142305 4.38 MB/sec execute 73 sec latency 31.695 ms 2024-04-13T03:39:33.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:32 smithi119 ceph-mon[35835]: pgmap v146: 129 pgs: 129 active+clean; 94 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-04-13T03:39:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:32 smithi016 ceph-mon[31772]: pgmap v146: 129 pgs: 129 active+clean; 94 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-04-13T03:39:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:32 smithi188 ceph-mon[36398]: pgmap v146: 129 pgs: 129 active+clean; 94 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-04-13T03:39:33.889 INFO:tasks.workunit.client.0.smithi016.stdout: 1 142633 4.32 MB/sec execute 74 sec latency 464.581 ms 2024-04-13T03:39:34.890 INFO:tasks.workunit.client.0.smithi016.stdout: 1 144246 4.35 MB/sec execute 75 sec latency 487.262 ms 2024-04-13T03:39:35.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:34 smithi119 ceph-mon[35835]: pgmap v147: 129 pgs: 129 active+clean; 94 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-13T03:39:35.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:34 smithi016 ceph-mon[31772]: pgmap v147: 129 pgs: 129 active+clean; 94 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-13T03:39:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:34 smithi188 ceph-mon[36398]: pgmap v147: 129 pgs: 129 active+clean; 94 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-13T03:39:35.890 INFO:tasks.workunit.client.0.smithi016.stdout: 1 145438 4.44 MB/sec execute 76 sec latency 31.925 ms 2024-04-13T03:39:36.890 INFO:tasks.workunit.client.0.smithi016.stdout: 1 146030 4.40 MB/sec execute 77 sec latency 23.588 ms 2024-04-13T03:39:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:36 smithi119 ceph-mon[35835]: pgmap v148: 129 pgs: 129 active+clean; 94 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-13T03:39:37.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:36 smithi016 ceph-mon[31772]: pgmap v148: 129 pgs: 129 active+clean; 94 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-13T03:39:37.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:36 smithi188 ceph-mon[36398]: pgmap v148: 129 pgs: 129 active+clean; 94 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-13T03:39:37.890 INFO:tasks.workunit.client.0.smithi016.stdout: 1 147127 4.41 MB/sec execute 78 sec latency 30.952 ms 2024-04-13T03:39:38.890 INFO:tasks.workunit.client.0.smithi016.stdout: 1 148646 4.46 MB/sec execute 79 sec latency 20.175 ms 2024-04-13T03:39:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:38 smithi119 ceph-mon[35835]: pgmap v149: 129 pgs: 129 active+clean; 94 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 1.9 MiB/s wr, 196 op/s 2024-04-13T03:39:39.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:38 smithi016 ceph-mon[31772]: pgmap v149: 129 pgs: 129 active+clean; 94 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 1.9 MiB/s wr, 196 op/s 2024-04-13T03:39:39.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:38 smithi188 ceph-mon[36398]: pgmap v149: 129 pgs: 129 active+clean; 94 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 1.9 MiB/s wr, 196 op/s 2024-04-13T03:39:39.890 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149412 4.47 MB/sec execute 80 sec latency 19.635 ms 2024-04-13T03:39:40.891 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149723 4.42 MB/sec execute 81 sec latency 525.479 ms 2024-04-13T03:39:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:40 smithi119 ceph-mon[35835]: pgmap v150: 129 pgs: 129 active+clean; 94 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-13T03:39:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:40 smithi016 ceph-mon[31772]: pgmap v150: 129 pgs: 129 active+clean; 94 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-13T03:39:41.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:40 smithi188 ceph-mon[36398]: pgmap v150: 129 pgs: 129 active+clean; 94 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-13T03:39:41.891 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149723 4.37 MB/sec execute 82 sec latency 1525.612 ms 2024-04-13T03:39:42.891 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149723 4.32 MB/sec execute 83 sec latency 2525.762 ms 2024-04-13T03:39:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:42 smithi119 ceph-mon[35835]: pgmap v151: 129 pgs: 129 active+clean; 97 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 199 op/s 2024-04-13T03:39:43.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:42 smithi188 ceph-mon[36398]: pgmap v151: 129 pgs: 129 active+clean; 97 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 199 op/s 2024-04-13T03:39:43.186 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:42 smithi016 ceph-mon[31772]: pgmap v151: 129 pgs: 129 active+clean; 97 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 199 op/s 2024-04-13T03:39:43.891 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149723 4.26 MB/sec execute 84 sec latency 3525.866 ms 2024-04-13T03:39:44.891 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149791 4.22 MB/sec execute 85 sec latency 3549.491 ms 2024-04-13T03:39:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:44 smithi016 ceph-mon[31772]: pgmap v152: 129 pgs: 129 active+clean; 97 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 886 KiB/s wr, 123 op/s 2024-04-13T03:39:45.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:44 smithi188 ceph-mon[36398]: pgmap v152: 129 pgs: 129 active+clean; 97 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 886 KiB/s wr, 123 op/s 2024-04-13T03:39:45.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:44 smithi119 ceph-mon[35835]: pgmap v152: 129 pgs: 129 active+clean; 97 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 886 KiB/s wr, 123 op/s 2024-04-13T03:39:45.891 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149791 4.17 MB/sec execute 86 sec latency 1875.204 ms 2024-04-13T03:39:46.891 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149791 4.12 MB/sec execute 87 sec latency 2875.353 ms 2024-04-13T03:39:47.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:46 smithi016 ceph-mon[31772]: pgmap v153: 129 pgs: 129 active+clean; 97 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 886 KiB/s wr, 123 op/s 2024-04-13T03:39:47.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:46 smithi188 ceph-mon[36398]: pgmap v153: 129 pgs: 129 active+clean; 97 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 886 KiB/s wr, 123 op/s 2024-04-13T03:39:47.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:46 smithi119 ceph-mon[35835]: pgmap v153: 129 pgs: 129 active+clean; 97 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 886 KiB/s wr, 123 op/s 2024-04-13T03:39:47.892 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149791 4.07 MB/sec execute 88 sec latency 3875.489 ms 2024-04-13T03:39:48.892 INFO:tasks.workunit.client.0.smithi016.stdout: 1 149791 4.03 MB/sec execute 89 sec latency 4875.656 ms 2024-04-13T03:39:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:48 smithi016 ceph-mon[31772]: pgmap v154: 129 pgs: 129 active+clean; 98 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.0 MiB/s wr, 141 op/s 2024-04-13T03:39:49.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:48 smithi188 ceph-mon[36398]: pgmap v154: 129 pgs: 129 active+clean; 98 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.0 MiB/s wr, 141 op/s 2024-04-13T03:39:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:48 smithi119 ceph-mon[35835]: pgmap v154: 129 pgs: 129 active+clean; 98 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.0 MiB/s wr, 141 op/s 2024-04-13T03:39:49.892 INFO:tasks.workunit.client.0.smithi016.stdout: 1 151360 4.05 MB/sec execute 90 sec latency 4897.577 ms 2024-04-13T03:39:50.892 INFO:tasks.workunit.client.0.smithi016.stdout: 1 152619 4.13 MB/sec execute 91 sec latency 25.949 ms 2024-04-13T03:39:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:50 smithi016 ceph-mon[31772]: pgmap v155: 129 pgs: 129 active+clean; 98 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 479 KiB/s wr, 70 op/s 2024-04-13T03:39:51.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:50 smithi188 ceph-mon[36398]: pgmap v155: 129 pgs: 129 active+clean; 98 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 479 KiB/s wr, 70 op/s 2024-04-13T03:39:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:50 smithi119 ceph-mon[35835]: pgmap v155: 129 pgs: 129 active+clean; 98 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 479 KiB/s wr, 70 op/s 2024-04-13T03:39:51.892 INFO:tasks.workunit.client.0.smithi016.stdout: 1 153217 4.10 MB/sec execute 92 sec latency 105.757 ms 2024-04-13T03:39:52.892 INFO:tasks.workunit.client.0.smithi016.stdout: 1 153217 4.06 MB/sec execute 93 sec latency 1105.879 ms 2024-04-13T03:39:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:52 smithi016 ceph-mon[31772]: pgmap v156: 129 pgs: 129 active+clean; 98 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-13T03:39:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:52 smithi188 ceph-mon[36398]: pgmap v156: 129 pgs: 129 active+clean; 98 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-13T03:39:53.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:52 smithi119 ceph-mon[35835]: pgmap v156: 129 pgs: 129 active+clean; 98 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 125 op/s 2024-04-13T03:39:53.893 INFO:tasks.workunit.client.0.smithi016.stdout: 1 153217 4.01 MB/sec execute 94 sec latency 2106.036 ms 2024-04-13T03:39:54.893 INFO:tasks.workunit.client.0.smithi016.stdout: 1 154885 4.04 MB/sec execute 95 sec latency 2128.751 ms 2024-04-13T03:39:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:54 smithi016 ceph-mon[31772]: pgmap v157: 129 pgs: 129 active+clean; 98 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 820 KiB/s wr, 72 op/s 2024-04-13T03:39:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:54 smithi188 ceph-mon[36398]: pgmap v157: 129 pgs: 129 active+clean; 98 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 820 KiB/s wr, 72 op/s 2024-04-13T03:39:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:54 smithi119 ceph-mon[35835]: pgmap v157: 129 pgs: 129 active+clean; 98 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 820 KiB/s wr, 72 op/s 2024-04-13T03:39:55.892 INFO:tasks.workunit.client.0.smithi016.stdout: 1 156154 4.11 MB/sec execute 96 sec latency 28.644 ms 2024-04-13T03:39:56.893 INFO:tasks.workunit.client.0.smithi016.stdout: 1 156814 4.09 MB/sec execute 97 sec latency 14.254 ms 2024-04-13T03:39:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:56 smithi188 ceph-mon[36398]: pgmap v158: 129 pgs: 129 active+clean; 98 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 820 KiB/s wr, 72 op/s 2024-04-13T03:39:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:56 smithi119 ceph-mon[35835]: pgmap v158: 129 pgs: 129 active+clean; 98 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 820 KiB/s wr, 72 op/s 2024-04-13T03:39:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:56 smithi016 ceph-mon[31772]: pgmap v158: 129 pgs: 129 active+clean; 98 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 820 KiB/s wr, 72 op/s 2024-04-13T03:39:57.893 INFO:tasks.workunit.client.0.smithi016.stdout: 1 156815 4.04 MB/sec execute 98 sec latency 995.340 ms 2024-04-13T03:39:58.893 INFO:tasks.workunit.client.0.smithi016.stdout: 1 156815 4.00 MB/sec execute 99 sec latency 1995.504 ms 2024-04-13T03:39:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:58 smithi188 ceph-mon[36398]: pgmap v159: 129 pgs: 129 active+clean; 101 MiB data, 901 MiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-13T03:39:59.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:58 smithi119 ceph-mon[35835]: pgmap v159: 129 pgs: 129 active+clean; 101 MiB data, 901 MiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-13T03:39:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:58 smithi016 ceph-mon[31772]: pgmap v159: 129 pgs: 129 active+clean; 101 MiB data, 901 MiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-13T03:39:59.893 INFO:tasks.workunit.client.0.smithi016.stdout: 1 158502 4.03 MB/sec execute 100 sec latency 2014.772 ms 2024-04-13T03:40:00.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:39:59 smithi119 ceph-mon[35835]: pgmap v160: 129 pgs: 129 active+clean; 101 MiB data, 901 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 965 KiB/s wr, 106 op/s 2024-04-13T03:40:00.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:39:59 smithi016 ceph-mon[31772]: pgmap v160: 129 pgs: 129 active+clean; 101 MiB data, 901 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 965 KiB/s wr, 106 op/s 2024-04-13T03:40:00.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:39:59 smithi188 ceph-mon[36398]: pgmap v160: 129 pgs: 129 active+clean; 101 MiB data, 901 MiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 965 KiB/s wr, 106 op/s 2024-04-13T03:40:00.893 INFO:tasks.workunit.client.0.smithi016.stdout: 1 159334 4.06 MB/sec execute 101 sec latency 76.707 ms 2024-04-13T03:40:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:00 smithi119 ceph-mon[35835]: overall HEALTH_OK 2024-04-13T03:40:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:00 smithi016 ceph-mon[31772]: overall HEALTH_OK 2024-04-13T03:40:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:00 smithi188 ceph-mon[36398]: overall HEALTH_OK 2024-04-13T03:40:01.893 INFO:tasks.workunit.client.0.smithi016.stdout: 1 159837 4.07 MB/sec execute 102 sec latency 82.771 ms 2024-04-13T03:40:02.221 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:01 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:40:02.222 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:01 smithi119 ceph-mon[35835]: pgmap v161: 129 pgs: 129 active+clean; 104 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.6 MiB/s wr, 184 op/s 2024-04-13T03:40:02.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:01 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:40:02.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:01 smithi016 ceph-mon[31772]: pgmap v161: 129 pgs: 129 active+clean; 104 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.6 MiB/s wr, 184 op/s 2024-04-13T03:40:02.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:01 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:40:02.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:01 smithi188 ceph-mon[36398]: pgmap v161: 129 pgs: 129 active+clean; 104 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.6 MiB/s wr, 184 op/s 2024-04-13T03:40:02.893 INFO:tasks.workunit.client.0.smithi016.stdout: 1 160306 4.03 MB/sec execute 103 sec latency 212.200 ms 2024-04-13T03:40:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:02 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:40:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:02 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:40:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:02 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:40:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:02 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:40:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:02 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:40:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:02 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:40:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:02 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:40:03.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:02 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:40:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:02 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:40:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:02 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:40:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:02 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:40:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:02 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:40:03.894 INFO:tasks.workunit.client.0.smithi016.stdout: 1 160306 3.99 MB/sec execute 104 sec latency 1212.298 ms 2024-04-13T03:40:04.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:03 smithi119 ceph-mon[35835]: pgmap v162: 129 pgs: 129 active+clean; 104 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.0 MiB/s wr, 129 op/s 2024-04-13T03:40:04.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:03 smithi016 ceph-mon[31772]: pgmap v162: 129 pgs: 129 active+clean; 104 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.0 MiB/s wr, 129 op/s 2024-04-13T03:40:04.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:03 smithi188 ceph-mon[36398]: pgmap v162: 129 pgs: 129 active+clean; 104 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.0 MiB/s wr, 129 op/s 2024-04-13T03:40:04.894 INFO:tasks.workunit.client.0.smithi016.stdout: 1 161683 4.01 MB/sec execute 105 sec latency 1237.627 ms 2024-04-13T03:40:05.894 INFO:tasks.workunit.client.0.smithi016.stdout: 1 163003 4.05 MB/sec execute 106 sec latency 57.829 ms 2024-04-13T03:40:06.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:06 smithi188 ceph-mon[36398]: pgmap v163: 129 pgs: 129 active+clean; 104 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.0 MiB/s wr, 129 op/s 2024-04-13T03:40:06.894 INFO:tasks.workunit.client.0.smithi016.stdout: 1 163690 4.06 MB/sec execute 107 sec latency 27.522 ms 2024-04-13T03:40:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:06 smithi119 ceph-mon[35835]: pgmap v163: 129 pgs: 129 active+clean; 104 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.0 MiB/s wr, 129 op/s 2024-04-13T03:40:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:06 smithi016 ceph-mon[31772]: pgmap v163: 129 pgs: 129 active+clean; 104 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 1.0 MiB/s wr, 129 op/s 2024-04-13T03:40:07.894 INFO:tasks.workunit.client.0.smithi016.stdout: 1 164689 4.07 MB/sec execute 108 sec latency 17.206 ms 2024-04-13T03:40:08.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:08 smithi188 ceph-mon[36398]: pgmap v164: 129 pgs: 129 active+clean; 106 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 1.8 MiB/s wr, 205 op/s 2024-04-13T03:40:08.894 INFO:tasks.workunit.client.0.smithi016.stdout: 1 166172 4.10 MB/sec execute 109 sec latency 23.953 ms 2024-04-13T03:40:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:08 smithi119 ceph-mon[35835]: pgmap v164: 129 pgs: 129 active+clean; 106 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 1.8 MiB/s wr, 205 op/s 2024-04-13T03:40:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:08 smithi016 ceph-mon[31772]: pgmap v164: 129 pgs: 129 active+clean; 106 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 1.8 MiB/s wr, 205 op/s 2024-04-13T03:40:09.894 INFO:tasks.workunit.client.0.smithi016.stdout: 1 167098 4.12 MB/sec execute 110 sec latency 33.011 ms 2024-04-13T03:40:10.894 INFO:tasks.workunit.client.0.smithi016.stdout: 1 167523 4.09 MB/sec execute 111 sec latency 256.437 ms 2024-04-13T03:40:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:10 smithi119 ceph-mon[35835]: pgmap v165: 129 pgs: 129 active+clean; 106 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-13T03:40:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:10 smithi016 ceph-mon[31772]: pgmap v165: 129 pgs: 129 active+clean; 106 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-13T03:40:11.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:10 smithi188 ceph-mon[36398]: pgmap v165: 129 pgs: 129 active+clean; 106 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-13T03:40:11.894 INFO:tasks.workunit.client.0.smithi016.stdout: 1 167523 4.05 MB/sec execute 112 sec latency 1256.539 ms 2024-04-13T03:40:12.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 167523 4.02 MB/sec execute 113 sec latency 2256.676 ms 2024-04-13T03:40:12.907 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:12 smithi016 ceph-mon[31772]: pgmap v166: 129 pgs: 129 active+clean; 111 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-04-13T03:40:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:12 smithi119 ceph-mon[35835]: pgmap v166: 129 pgs: 129 active+clean; 111 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-04-13T03:40:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:12 smithi188 ceph-mon[36398]: pgmap v166: 129 pgs: 129 active+clean; 111 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 2.1 MiB/s wr, 217 op/s 2024-04-13T03:40:13.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 167523 3.98 MB/sec execute 114 sec latency 3256.785 ms 2024-04-13T03:40:14.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 169079 4.00 MB/sec execute 115 sec latency 3285.317 ms 2024-04-13T03:40:15.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:14 smithi119 ceph-mon[35835]: pgmap v167: 129 pgs: 129 active+clean; 111 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-13T03:40:15.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:14 smithi016 ceph-mon[31772]: pgmap v167: 129 pgs: 129 active+clean; 111 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-13T03:40:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:14 smithi188 ceph-mon[36398]: pgmap v167: 129 pgs: 129 active+clean; 111 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-13T03:40:15.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 170106 4.03 MB/sec execute 116 sec latency 25.594 ms 2024-04-13T03:40:16.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 170743 4.04 MB/sec execute 117 sec latency 17.731 ms 2024-04-13T03:40:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:16 smithi119 ceph-mon[35835]: pgmap v168: 129 pgs: 129 active+clean; 111 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-13T03:40:17.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:16 smithi016 ceph-mon[31772]: pgmap v168: 129 pgs: 129 active+clean; 111 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-13T03:40:17.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:16 smithi188 ceph-mon[36398]: pgmap v168: 129 pgs: 129 active+clean; 111 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.4 MiB/s wr, 139 op/s 2024-04-13T03:40:17.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 171063 4.01 MB/sec execute 118 sec latency 560.605 ms 2024-04-13T03:40:18.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 171063 3.97 MB/sec execute 119 sec latency 1560.736 ms 2024-04-13T03:40:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:18 smithi119 ceph-mon[35835]: pgmap v169: 129 pgs: 129 active+clean; 110 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 1.6 MiB/s wr, 182 op/s 2024-04-13T03:40:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:18 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:40:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:18 smithi016 ceph-mon[31772]: pgmap v169: 129 pgs: 129 active+clean; 110 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 1.6 MiB/s wr, 182 op/s 2024-04-13T03:40:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:18 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:40:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:18 smithi188 ceph-mon[36398]: pgmap v169: 129 pgs: 129 active+clean; 110 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 1.6 MiB/s wr, 182 op/s 2024-04-13T03:40:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:18 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:40:19.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 172628 4.00 MB/sec execute 120 sec latency 1580.100 ms 2024-04-13T03:40:20.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 173802 4.05 MB/sec execute 121 sec latency 17.907 ms 2024-04-13T03:40:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:20 smithi119 ceph-mon[35835]: pgmap v170: 129 pgs: 129 active+clean; 110 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 914 KiB/s wr, 107 op/s 2024-04-13T03:40:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:20 smithi016 ceph-mon[31772]: pgmap v170: 129 pgs: 129 active+clean; 110 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 914 KiB/s wr, 107 op/s 2024-04-13T03:40:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:20 smithi188 ceph-mon[36398]: pgmap v170: 129 pgs: 129 active+clean; 110 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 914 KiB/s wr, 107 op/s 2024-04-13T03:40:21.896 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174420 4.03 MB/sec execute 122 sec latency 33.458 ms 2024-04-13T03:40:22.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:22 smithi016 ceph-mon[31772]: pgmap v171: 129 pgs: 129 active+clean; 114 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-04-13T03:40:22.895 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174489 4.00 MB/sec execute 123 sec latency 958.117 ms 2024-04-13T03:40:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:22 smithi119 ceph-mon[35835]: pgmap v171: 129 pgs: 129 active+clean; 114 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-04-13T03:40:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:22 smithi188 ceph-mon[36398]: pgmap v171: 129 pgs: 129 active+clean; 114 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-04-13T03:40:23.896 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174489 3.97 MB/sec execute 124 sec latency 1958.193 ms 2024-04-13T03:40:24.896 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174613 3.94 MB/sec execute 125 sec latency 1982.183 ms 2024-04-13T03:40:24.925 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:24 smithi188 ceph-mon[36398]: pgmap v172: 129 pgs: 129 active+clean; 114 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 842 KiB/s wr, 96 op/s 2024-04-13T03:40:25.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:24 smithi119 ceph-mon[35835]: pgmap v172: 129 pgs: 129 active+clean; 114 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 842 KiB/s wr, 96 op/s 2024-04-13T03:40:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:24 smithi016 ceph-mon[31772]: pgmap v172: 129 pgs: 129 active+clean; 114 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 842 KiB/s wr, 96 op/s 2024-04-13T03:40:25.896 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174613 3.91 MB/sec execute 126 sec latency 1786.222 ms 2024-04-13T03:40:26.897 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174613 3.88 MB/sec execute 127 sec latency 2786.811 ms 2024-04-13T03:40:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:26 smithi119 ceph-mon[35835]: pgmap v173: 129 pgs: 129 active+clean; 114 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 842 KiB/s wr, 96 op/s 2024-04-13T03:40:27.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:26 smithi016 ceph-mon[31772]: pgmap v173: 129 pgs: 129 active+clean; 114 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 842 KiB/s wr, 96 op/s 2024-04-13T03:40:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:26 smithi188 ceph-mon[36398]: pgmap v173: 129 pgs: 129 active+clean; 114 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 842 KiB/s wr, 96 op/s 2024-04-13T03:40:27.897 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174613 3.84 MB/sec execute 128 sec latency 3786.932 ms 2024-04-13T03:40:28.897 INFO:tasks.workunit.client.0.smithi016.stdout: 1 174613 3.81 MB/sec execute 129 sec latency 4787.084 ms 2024-04-13T03:40:29.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:28 smithi119 ceph-mon[35835]: pgmap v174: 129 pgs: 129 active+clean; 117 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-13T03:40:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:28 smithi016 ceph-mon[31772]: pgmap v174: 129 pgs: 129 active+clean; 117 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-13T03:40:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:28 smithi188 ceph-mon[36398]: pgmap v174: 129 pgs: 129 active+clean; 117 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-13T03:40:29.897 INFO:tasks.workunit.client.0.smithi016.stdout: 1 176191 3.84 MB/sec execute 130 sec latency 4805.646 ms 2024-04-13T03:40:30.897 INFO:tasks.workunit.client.0.smithi016.stdout: 1 177443 3.89 MB/sec execute 131 sec latency 23.070 ms 2024-04-13T03:40:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:30 smithi119 ceph-mon[35835]: pgmap v175: 129 pgs: 129 active+clean; 117 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 901 KiB/s wr, 79 op/s 2024-04-13T03:40:31.107 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:30 smithi016 ceph-mon[31772]: pgmap v175: 129 pgs: 129 active+clean; 117 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 901 KiB/s wr, 79 op/s 2024-04-13T03:40:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:30 smithi188 ceph-mon[36398]: pgmap v175: 129 pgs: 129 active+clean; 117 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 901 KiB/s wr, 79 op/s 2024-04-13T03:40:31.898 INFO:tasks.workunit.client.0.smithi016.stdout: 1 178042 3.87 MB/sec execute 132 sec latency 21.861 ms 2024-04-13T03:40:32.898 INFO:tasks.workunit.client.0.smithi016.stdout: 1 179720 3.89 MB/sec execute 133 sec latency 19.204 ms 2024-04-13T03:40:33.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:32 smithi119 ceph-mon[35835]: pgmap v176: 129 pgs: 129 active+clean; 117 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-04-13T03:40:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:32 smithi016 ceph-mon[31772]: pgmap v176: 129 pgs: 129 active+clean; 117 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-04-13T03:40:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:32 smithi188 ceph-mon[36398]: pgmap v176: 129 pgs: 129 active+clean; 117 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 129 op/s 2024-04-13T03:40:33.898 INFO:tasks.workunit.client.0.smithi016.stdout: 1 181027 3.94 MB/sec execute 134 sec latency 14.494 ms 2024-04-13T03:40:34.898 INFO:tasks.workunit.client.0.smithi016.stdout: 1 181608 3.93 MB/sec execute 135 sec latency 14.663 ms 2024-04-13T03:40:35.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:34 smithi119 ceph-mon[35835]: pgmap v177: 129 pgs: 129 active+clean; 117 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 744 KiB/s wr, 76 op/s 2024-04-13T03:40:35.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:34 smithi016 ceph-mon[31772]: pgmap v177: 129 pgs: 129 active+clean; 117 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 744 KiB/s wr, 76 op/s 2024-04-13T03:40:35.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:34 smithi188 ceph-mon[36398]: pgmap v177: 129 pgs: 129 active+clean; 117 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 744 KiB/s wr, 76 op/s 2024-04-13T03:40:35.898 INFO:tasks.workunit.client.0.smithi016.stdout: 1 183204 3.95 MB/sec execute 136 sec latency 17.520 ms 2024-04-13T03:40:36.898 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184288 3.97 MB/sec execute 137 sec latency 37.415 ms 2024-04-13T03:40:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:36 smithi119 ceph-mon[35835]: pgmap v178: 129 pgs: 129 active+clean; 117 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 744 KiB/s wr, 76 op/s 2024-04-13T03:40:37.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:36 smithi016 ceph-mon[31772]: pgmap v178: 129 pgs: 129 active+clean; 117 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 744 KiB/s wr, 76 op/s 2024-04-13T03:40:37.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:36 smithi188 ceph-mon[36398]: pgmap v178: 129 pgs: 129 active+clean; 117 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 744 KiB/s wr, 76 op/s 2024-04-13T03:40:37.899 INFO:tasks.workunit.client.0.smithi016.stdout: 1 184991 3.98 MB/sec execute 138 sec latency 27.362 ms 2024-04-13T03:40:38.898 INFO:tasks.workunit.client.0.smithi016.stdout: 1 185178 3.95 MB/sec execute 139 sec latency 772.832 ms 2024-04-13T03:40:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:38 smithi119 ceph-mon[35835]: pgmap v179: 129 pgs: 129 active+clean; 121 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-04-13T03:40:39.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:38 smithi016 ceph-mon[31772]: pgmap v179: 129 pgs: 129 active+clean; 121 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-04-13T03:40:39.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:38 smithi188 ceph-mon[36398]: pgmap v179: 129 pgs: 129 active+clean; 121 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 154 op/s 2024-04-13T03:40:39.899 INFO:tasks.workunit.client.0.smithi016.stdout: 1 185246 3.92 MB/sec execute 140 sec latency 877.667 ms 2024-04-13T03:40:40.899 INFO:tasks.workunit.client.0.smithi016.stdout: 1 185246 3.90 MB/sec execute 141 sec latency 1877.827 ms 2024-04-13T03:40:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:40 smithi119 ceph-mon[35835]: pgmap v180: 129 pgs: 129 active+clean; 121 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-13T03:40:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:40 smithi016 ceph-mon[31772]: pgmap v180: 129 pgs: 129 active+clean; 121 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-13T03:40:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:40 smithi188 ceph-mon[36398]: pgmap v180: 129 pgs: 129 active+clean; 121 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 127 op/s 2024-04-13T03:40:41.899 INFO:tasks.workunit.client.0.smithi016.stdout: 1 185246 3.87 MB/sec execute 142 sec latency 2878.286 ms 2024-04-13T03:40:42.899 INFO:tasks.workunit.client.0.smithi016.stdout: 1 185246 3.84 MB/sec execute 143 sec latency 3878.514 ms 2024-04-13T03:40:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:42 smithi119 ceph-mon[35835]: pgmap v181: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 163 op/s 2024-04-13T03:40:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:42 smithi016 ceph-mon[31772]: pgmap v181: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 163 op/s 2024-04-13T03:40:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:42 smithi188 ceph-mon[36398]: pgmap v181: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 163 op/s 2024-04-13T03:40:43.900 INFO:tasks.workunit.client.0.smithi016.stdout: 1 185246 3.82 MB/sec execute 144 sec latency 4878.646 ms 2024-04-13T03:40:44.900 INFO:tasks.workunit.client.0.smithi016.stdout: 1 186833 3.84 MB/sec execute 145 sec latency 4897.691 ms 2024-04-13T03:40:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:44 smithi119 ceph-mon[35835]: pgmap v182: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-13T03:40:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:44 smithi016 ceph-mon[31772]: pgmap v182: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-13T03:40:45.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:44 smithi188 ceph-mon[36398]: pgmap v182: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-13T03:40:45.900 INFO:tasks.workunit.client.0.smithi016.stdout: 1 188074 3.88 MB/sec execute 146 sec latency 23.934 ms 2024-04-13T03:40:46.900 INFO:tasks.workunit.client.0.smithi016.stdout: 1 188614 3.87 MB/sec execute 147 sec latency 25.160 ms 2024-04-13T03:40:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:46 smithi119 ceph-mon[35835]: pgmap v183: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-13T03:40:47.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:46 smithi016 ceph-mon[31772]: pgmap v183: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-13T03:40:47.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:46 smithi188 ceph-mon[36398]: pgmap v183: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.1 MiB/s wr, 113 op/s 2024-04-13T03:40:47.900 INFO:tasks.workunit.client.0.smithi016.stdout: 1 188672 3.84 MB/sec execute 148 sec latency 975.413 ms 2024-04-13T03:40:48.900 INFO:tasks.workunit.client.0.smithi016.stdout: 1 188672 3.81 MB/sec execute 149 sec latency 1975.564 ms 2024-04-13T03:40:49.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:48 smithi119 ceph-mon[35835]: pgmap v184: 129 pgs: 129 active+clean; 122 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-13T03:40:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:48 smithi016 ceph-mon[31772]: pgmap v184: 129 pgs: 129 active+clean; 122 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-13T03:40:49.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:48 smithi188 ceph-mon[36398]: pgmap v184: 129 pgs: 129 active+clean; 122 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-04-13T03:40:49.900 INFO:tasks.workunit.client.0.smithi016.stdout: 1 189867 3.82 MB/sec execute 150 sec latency 1996.200 ms 2024-04-13T03:40:50.901 INFO:tasks.workunit.client.0.smithi016.stdout: 1 191361 3.86 MB/sec execute 151 sec latency 15.117 ms 2024-04-13T03:40:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:50 smithi119 ceph-mon[35835]: pgmap v185: 129 pgs: 129 active+clean; 122 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 633 KiB/s wr, 79 op/s 2024-04-13T03:40:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:50 smithi016 ceph-mon[31772]: pgmap v185: 129 pgs: 129 active+clean; 122 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 633 KiB/s wr, 79 op/s 2024-04-13T03:40:51.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:50 smithi188 ceph-mon[36398]: pgmap v185: 129 pgs: 129 active+clean; 122 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 633 KiB/s wr, 79 op/s 2024-04-13T03:40:51.901 INFO:tasks.workunit.client.0.smithi016.stdout: 1 192074 3.86 MB/sec execute 152 sec latency 43.228 ms 2024-04-13T03:40:52.901 INFO:tasks.workunit.client.0.smithi016.stdout: 1 192337 3.84 MB/sec execute 153 sec latency 631.675 ms 2024-04-13T03:40:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:52 smithi119 ceph-mon[35835]: pgmap v186: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.1 MiB/s wr, 130 op/s 2024-04-13T03:40:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:52 smithi016 ceph-mon[31772]: pgmap v186: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.1 MiB/s wr, 130 op/s 2024-04-13T03:40:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:52 smithi188 ceph-mon[36398]: pgmap v186: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 1.1 MiB/s wr, 130 op/s 2024-04-13T03:40:53.901 INFO:tasks.workunit.client.0.smithi016.stdout: 1 192337 3.82 MB/sec execute 154 sec latency 1631.770 ms 2024-04-13T03:40:54.901 INFO:tasks.workunit.client.0.smithi016.stdout: 1 193937 3.83 MB/sec execute 155 sec latency 1649.136 ms 2024-04-13T03:40:55.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:54 smithi119 ceph-mon[35835]: pgmap v187: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 703 KiB/s wr, 94 op/s 2024-04-13T03:40:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:54 smithi016 ceph-mon[31772]: pgmap v187: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 703 KiB/s wr, 94 op/s 2024-04-13T03:40:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:54 smithi188 ceph-mon[36398]: pgmap v187: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 703 KiB/s wr, 94 op/s 2024-04-13T03:40:55.901 INFO:tasks.workunit.client.0.smithi016.stdout: 1 195172 3.88 MB/sec execute 156 sec latency 15.855 ms 2024-04-13T03:40:56.901 INFO:tasks.workunit.client.0.smithi016.stdout: 1 195716 3.86 MB/sec execute 157 sec latency 17.512 ms 2024-04-13T03:40:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:56 smithi119 ceph-mon[35835]: pgmap v188: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 703 KiB/s wr, 94 op/s 2024-04-13T03:40:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:56 smithi016 ceph-mon[31772]: pgmap v188: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 703 KiB/s wr, 94 op/s 2024-04-13T03:40:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:56 smithi188 ceph-mon[36398]: pgmap v188: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 703 KiB/s wr, 94 op/s 2024-04-13T03:40:57.902 INFO:tasks.workunit.client.0.smithi016.stdout: 1 195887 3.84 MB/sec execute 158 sec latency 784.182 ms 2024-04-13T03:40:58.902 INFO:tasks.workunit.client.0.smithi016.stdout: 1 195887 3.82 MB/sec execute 159 sec latency 1784.308 ms 2024-04-13T03:40:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:40:58 smithi119 ceph-mon[35835]: pgmap v189: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-13T03:40:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:40:58 smithi016 ceph-mon[31772]: pgmap v189: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-13T03:40:59.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:40:58 smithi188 ceph-mon[36398]: pgmap v189: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.3 MiB/s wr, 151 op/s 2024-04-13T03:40:59.902 INFO:tasks.workunit.client.0.smithi016.stdout: 1 197393 3.83 MB/sec execute 160 sec latency 1799.053 ms 2024-04-13T03:41:00.902 INFO:tasks.workunit.client.0.smithi016.stdout: 1 197702 3.82 MB/sec execute 161 sec latency 578.655 ms 2024-04-13T03:41:01.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:00 smithi119 ceph-mon[35835]: pgmap v190: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-13T03:41:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:00 smithi016 ceph-mon[31772]: pgmap v190: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-13T03:41:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:00 smithi188 ceph-mon[36398]: pgmap v190: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 1.1 MiB/s wr, 107 op/s 2024-04-13T03:41:01.902 INFO:tasks.workunit.client.0.smithi016.stdout: 1 197702 3.79 MB/sec execute 162 sec latency 1578.757 ms 2024-04-13T03:41:02.902 INFO:tasks.workunit.client.0.smithi016.stdout: 1 197702 3.77 MB/sec execute 163 sec latency 2578.904 ms 2024-04-13T03:41:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:02 smithi119 ceph-mon[35835]: pgmap v191: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-13T03:41:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:02 smithi016 ceph-mon[31772]: pgmap v191: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-13T03:41:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:02 smithi188 ceph-mon[36398]: pgmap v191: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-13T03:41:03.903 INFO:tasks.workunit.client.0.smithi016.stdout: 1 197988 3.77 MB/sec execute 164 sec latency 2793.517 ms 2024-04-13T03:41:04.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:03 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:41:04.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:03 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:41:04.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:03 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:41:04.902 INFO:tasks.workunit.client.0.smithi016.stdout: 1 198104 3.75 MB/sec execute 165 sec latency 1509.337 ms 2024-04-13T03:41:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:04 smithi119 ceph-mon[35835]: pgmap v192: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 826 KiB/s wr, 90 op/s 2024-04-13T03:41:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:04 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:41:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:04 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:41:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:04 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:41:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:04 smithi016 ceph-mon[31772]: pgmap v192: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 826 KiB/s wr, 90 op/s 2024-04-13T03:41:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:04 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:41:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:04 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:41:05.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:04 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:41:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:04 smithi188 ceph-mon[36398]: pgmap v192: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 826 KiB/s wr, 90 op/s 2024-04-13T03:41:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:04 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:41:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:04 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:41:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:04 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:41:05.902 INFO:tasks.workunit.client.0.smithi016.stdout: 1 198352 3.74 MB/sec execute 166 sec latency 93.066 ms 2024-04-13T03:41:06.903 INFO:tasks.workunit.client.0.smithi016.stdout: 1 198959 3.74 MB/sec execute 167 sec latency 87.047 ms 2024-04-13T03:41:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:06 smithi119 ceph-mon[35835]: pgmap v193: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 826 KiB/s wr, 90 op/s 2024-04-13T03:41:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:06 smithi016 ceph-mon[31772]: pgmap v193: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 826 KiB/s wr, 90 op/s 2024-04-13T03:41:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:06 smithi188 ceph-mon[36398]: pgmap v193: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 826 KiB/s wr, 90 op/s 2024-04-13T03:41:07.904 INFO:tasks.workunit.client.0.smithi016.stdout: 1 199522 3.73 MB/sec execute 168 sec latency 69.332 ms 2024-04-13T03:41:08.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:07 smithi119 ceph-mon[35835]: osdmap e96: 12 total, 12 up, 12 in 2024-04-13T03:41:08.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:07 smithi016 ceph-mon[31772]: osdmap e96: 12 total, 12 up, 12 in 2024-04-13T03:41:08.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:07 smithi188 ceph-mon[36398]: osdmap e96: 12 total, 12 up, 12 in 2024-04-13T03:41:08.904 INFO:tasks.workunit.client.0.smithi016.stdout: 1 201077 3.74 MB/sec execute 169 sec latency 21.069 ms 2024-04-13T03:41:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:08 smithi119 ceph-mon[35835]: pgmap v195: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 767 KiB/s wr, 116 op/s 2024-04-13T03:41:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:08 smithi016 ceph-mon[31772]: pgmap v195: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 767 KiB/s wr, 116 op/s 2024-04-13T03:41:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:08 smithi188 ceph-mon[36398]: pgmap v195: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 767 KiB/s wr, 116 op/s 2024-04-13T03:41:09.904 INFO:tasks.workunit.client.0.smithi016.stdout: 1 202272 3.78 MB/sec execute 170 sec latency 27.242 ms 2024-04-13T03:41:10.904 INFO:tasks.workunit.client.0.smithi016.stdout: 1 202788 3.77 MB/sec execute 171 sec latency 16.956 ms 2024-04-13T03:41:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:10 smithi119 ceph-mon[35835]: pgmap v196: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 767 KiB/s wr, 116 op/s 2024-04-13T03:41:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:10 smithi016 ceph-mon[31772]: pgmap v196: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 767 KiB/s wr, 116 op/s 2024-04-13T03:41:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:10 smithi188 ceph-mon[36398]: pgmap v196: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 767 KiB/s wr, 116 op/s 2024-04-13T03:41:11.904 INFO:tasks.workunit.client.0.smithi016.stdout: 1 202976 3.75 MB/sec execute 172 sec latency 743.061 ms 2024-04-13T03:41:12.905 INFO:tasks.workunit.client.0.smithi016.stdout: 1 202976 3.73 MB/sec execute 173 sec latency 1743.266 ms 2024-04-13T03:41:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:12 smithi119 ceph-mon[35835]: pgmap v197: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.5 MiB/s wr, 167 op/s 2024-04-13T03:41:13.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:12 smithi016 ceph-mon[31772]: pgmap v197: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.5 MiB/s wr, 167 op/s 2024-04-13T03:41:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:12 smithi188 ceph-mon[36398]: pgmap v197: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.5 MiB/s wr, 167 op/s 2024-04-13T03:41:13.905 INFO:tasks.workunit.client.0.smithi016.stdout: 1 202976 3.71 MB/sec execute 174 sec latency 2743.479 ms 2024-04-13T03:41:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:14 smithi119 ceph-mon[35835]: osdmap e97: 12 total, 12 up, 12 in 2024-04-13T03:41:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:14 smithi119 ceph-mon[35835]: pgmap v199: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-04-13T03:41:14.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:14 smithi016 ceph-mon[31772]: osdmap e97: 12 total, 12 up, 12 in 2024-04-13T03:41:14.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:14 smithi016 ceph-mon[31772]: pgmap v199: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-04-13T03:41:14.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:14 smithi188 ceph-mon[36398]: osdmap e97: 12 total, 12 up, 12 in 2024-04-13T03:41:14.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:14 smithi188 ceph-mon[36398]: pgmap v199: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-04-13T03:41:14.905 INFO:tasks.workunit.client.0.smithi016.stdout: 1 204589 3.72 MB/sec execute 175 sec latency 2755.882 ms 2024-04-13T03:41:15.905 INFO:tasks.workunit.client.0.smithi016.stdout: 1 205803 3.76 MB/sec execute 176 sec latency 15.951 ms 2024-04-13T03:41:16.905 INFO:tasks.workunit.client.0.smithi016.stdout: 1 206303 3.75 MB/sec execute 177 sec latency 18.209 ms 2024-04-13T03:41:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:16 smithi119 ceph-mon[35835]: pgmap v200: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 1.2 MiB/s wr, 103 op/s 2024-04-13T03:41:17.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:16 smithi016 ceph-mon[31772]: pgmap v200: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 1.2 MiB/s wr, 103 op/s 2024-04-13T03:41:17.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:16 smithi188 ceph-mon[36398]: pgmap v200: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 1.2 MiB/s wr, 103 op/s 2024-04-13T03:41:17.905 INFO:tasks.workunit.client.0.smithi016.stdout: 1 206396 3.73 MB/sec execute 178 sec latency 917.352 ms 2024-04-13T03:41:18.906 INFO:tasks.workunit.client.0.smithi016.stdout: 1 206396 3.71 MB/sec execute 179 sec latency 1917.466 ms 2024-04-13T03:41:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:18 smithi119 ceph-mon[35835]: pgmap v201: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-13T03:41:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:18 smithi016 ceph-mon[31772]: pgmap v201: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-13T03:41:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:18 smithi188 ceph-mon[36398]: pgmap v201: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-13T03:41:19.906 INFO:tasks.workunit.client.0.smithi016.stdout: 1 207408 3.72 MB/sec execute 180 sec latency 1942.014 ms 2024-04-13T03:41:20.906 INFO:tasks.workunit.client.0.smithi016.stdout: 1 208749 3.74 MB/sec execute 181 sec latency 24.079 ms 2024-04-13T03:41:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:20 smithi119 ceph-mon[35835]: pgmap v202: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-13T03:41:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:20 smithi016 ceph-mon[31772]: pgmap v202: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-13T03:41:21.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:20 smithi188 ceph-mon[36398]: pgmap v202: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-04-13T03:41:21.908 INFO:tasks.workunit.client.0.smithi016.stdout: 1 209657 3.75 MB/sec execute 182 sec latency 18.503 ms 2024-04-13T03:41:22.907 INFO:tasks.workunit.client.0.smithi016.stdout: 1 209945 3.73 MB/sec execute 183 sec latency 430.718 ms 2024-04-13T03:41:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:22 smithi119 ceph-mon[35835]: pgmap v203: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:41:23.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:22 smithi016 ceph-mon[31772]: pgmap v203: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:41:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:22 smithi188 ceph-mon[36398]: pgmap v203: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:41:23.907 INFO:tasks.workunit.client.0.smithi016.stdout: 1 209945 3.71 MB/sec execute 184 sec latency 1430.855 ms 2024-04-13T03:41:24.908 INFO:tasks.workunit.client.0.smithi016.stdout: 1 210069 3.69 MB/sec execute 185 sec latency 1440.281 ms 2024-04-13T03:41:24.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:24 smithi188 ceph-mon[36398]: pgmap v204: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 975 KiB/s wr, 105 op/s 2024-04-13T03:41:25.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:24 smithi119 ceph-mon[35835]: pgmap v204: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 975 KiB/s wr, 105 op/s 2024-04-13T03:41:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:24 smithi016 ceph-mon[31772]: pgmap v204: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 975 KiB/s wr, 105 op/s 2024-04-13T03:41:25.908 INFO:tasks.workunit.client.0.smithi016.stdout: 1 210069 3.67 MB/sec execute 186 sec latency 1816.926 ms 2024-04-13T03:41:26.908 INFO:tasks.workunit.client.0.smithi016.stdout: 1 210069 3.65 MB/sec execute 187 sec latency 2817.041 ms 2024-04-13T03:41:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:26 smithi119 ceph-mon[35835]: pgmap v205: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 860 KiB/s wr, 93 op/s 2024-04-13T03:41:27.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:26 smithi016 ceph-mon[31772]: pgmap v205: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 860 KiB/s wr, 93 op/s 2024-04-13T03:41:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:26 smithi188 ceph-mon[36398]: pgmap v205: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 860 KiB/s wr, 93 op/s 2024-04-13T03:41:27.908 INFO:tasks.workunit.client.0.smithi016.stdout: 1 210069 3.63 MB/sec execute 188 sec latency 3817.177 ms 2024-04-13T03:41:28.909 INFO:tasks.workunit.client.0.smithi016.stdout: 1 210069 3.62 MB/sec execute 189 sec latency 4817.298 ms 2024-04-13T03:41:29.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:28 smithi119 ceph-mon[35835]: pgmap v206: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-13T03:41:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:28 smithi016 ceph-mon[31772]: pgmap v206: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-13T03:41:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:28 smithi188 ceph-mon[36398]: pgmap v206: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-04-13T03:41:29.908 INFO:tasks.workunit.client.0.smithi016.stdout: 1 211696 3.63 MB/sec execute 190 sec latency 4830.369 ms 2024-04-13T03:41:30.908 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213023 3.67 MB/sec execute 191 sec latency 19.572 ms 2024-04-13T03:41:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:30 smithi119 ceph-mon[35835]: pgmap v207: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 946 KiB/s wr, 88 op/s 2024-04-13T03:41:31.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:30 smithi016 ceph-mon[31772]: pgmap v207: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 946 KiB/s wr, 88 op/s 2024-04-13T03:41:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:30 smithi188 ceph-mon[36398]: pgmap v207: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 946 KiB/s wr, 88 op/s 2024-04-13T03:41:31.909 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213541 3.66 MB/sec execute 192 sec latency 89.440 ms 2024-04-13T03:41:32.909 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213541 3.64 MB/sec execute 193 sec latency 1089.542 ms 2024-04-13T03:41:33.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:32 smithi119 ceph-mon[35835]: pgmap v208: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-04-13T03:41:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:32 smithi016 ceph-mon[31772]: pgmap v208: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-04-13T03:41:33.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:32 smithi188 ceph-mon[36398]: pgmap v208: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-04-13T03:41:33.909 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213541 3.62 MB/sec execute 194 sec latency 2089.636 ms 2024-04-13T03:41:34.909 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213609 3.60 MB/sec execute 195 sec latency 2097.450 ms 2024-04-13T03:41:35.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:34 smithi119 ceph-mon[35835]: pgmap v209: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 935 KiB/s wr, 83 op/s 2024-04-13T03:41:35.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:34 smithi016 ceph-mon[31772]: pgmap v209: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 935 KiB/s wr, 83 op/s 2024-04-13T03:41:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:34 smithi188 ceph-mon[36398]: pgmap v209: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 935 KiB/s wr, 83 op/s 2024-04-13T03:41:35.909 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213609 3.58 MB/sec execute 196 sec latency 1909.172 ms 2024-04-13T03:41:36.909 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213609 3.57 MB/sec execute 197 sec latency 2909.330 ms 2024-04-13T03:41:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:36 smithi119 ceph-mon[35835]: pgmap v210: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 935 KiB/s wr, 83 op/s 2024-04-13T03:41:37.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:36 smithi016 ceph-mon[31772]: pgmap v210: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 935 KiB/s wr, 83 op/s 2024-04-13T03:41:37.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:36 smithi188 ceph-mon[36398]: pgmap v210: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 935 KiB/s wr, 83 op/s 2024-04-13T03:41:37.909 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213609 3.55 MB/sec execute 198 sec latency 3909.499 ms 2024-04-13T03:41:38.909 INFO:tasks.workunit.client.0.smithi016.stdout: 1 213609 3.53 MB/sec execute 199 sec latency 4909.661 ms 2024-04-13T03:41:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:38 smithi119 ceph-mon[35835]: pgmap v211: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-04-13T03:41:39.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:38 smithi016 ceph-mon[31772]: pgmap v211: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-04-13T03:41:39.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:38 smithi188 ceph-mon[36398]: pgmap v211: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-04-13T03:41:39.910 INFO:tasks.workunit.client.0.smithi016.stdout: 1 215316 3.55 MB/sec execute 200 sec latency 4917.856 ms 2024-04-13T03:41:40.910 INFO:tasks.workunit.client.0.smithi016.stdout: 1 216648 3.59 MB/sec execute 201 sec latency 19.096 ms 2024-04-13T03:41:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:40 smithi119 ceph-mon[35835]: pgmap v212: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 658 KiB/s wr, 70 op/s 2024-04-13T03:41:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:40 smithi016 ceph-mon[31772]: pgmap v212: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 658 KiB/s wr, 70 op/s 2024-04-13T03:41:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:40 smithi188 ceph-mon[36398]: pgmap v212: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 658 KiB/s wr, 70 op/s 2024-04-13T03:41:41.910 INFO:tasks.workunit.client.0.smithi016.stdout: 1 217245 3.57 MB/sec execute 202 sec latency 16.630 ms 2024-04-13T03:41:42.910 INFO:tasks.workunit.client.0.smithi016.stdout: 1 218858 3.59 MB/sec execute 203 sec latency 55.638 ms 2024-04-13T03:41:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:42 smithi119 ceph-mon[35835]: pgmap v213: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-04-13T03:41:43.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:42 smithi016 ceph-mon[31772]: pgmap v213: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-04-13T03:41:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:42 smithi188 ceph-mon[36398]: pgmap v213: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-04-13T03:41:43.910 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220054 3.62 MB/sec execute 204 sec latency 22.317 ms 2024-04-13T03:41:44.910 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220532 3.61 MB/sec execute 205 sec latency 21.577 ms 2024-04-13T03:41:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:44 smithi119 ceph-mon[35835]: pgmap v214: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 509 KiB/s wr, 62 op/s 2024-04-13T03:41:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:44 smithi016 ceph-mon[31772]: pgmap v214: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 509 KiB/s wr, 62 op/s 2024-04-13T03:41:45.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:44 smithi188 ceph-mon[36398]: pgmap v214: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 509 KiB/s wr, 62 op/s 2024-04-13T03:41:45.910 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220632 3.59 MB/sec execute 206 sec latency 911.666 ms 2024-04-13T03:41:46.910 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220632 3.58 MB/sec execute 207 sec latency 1911.799 ms 2024-04-13T03:41:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:46 smithi119 ceph-mon[35835]: pgmap v215: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 509 KiB/s wr, 62 op/s 2024-04-13T03:41:47.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:46 smithi016 ceph-mon[31772]: pgmap v215: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 509 KiB/s wr, 62 op/s 2024-04-13T03:41:47.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:46 smithi188 ceph-mon[36398]: pgmap v215: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 509 KiB/s wr, 62 op/s 2024-04-13T03:41:47.911 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220632 3.56 MB/sec execute 208 sec latency 2912.550 ms 2024-04-13T03:41:48.911 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220632 3.54 MB/sec execute 209 sec latency 3912.673 ms 2024-04-13T03:41:49.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:48 smithi119 ceph-mon[35835]: pgmap v216: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-13T03:41:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:48 smithi016 ceph-mon[31772]: pgmap v216: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-13T03:41:49.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:48 smithi188 ceph-mon[36398]: pgmap v216: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 1.1 MiB/s wr, 119 op/s 2024-04-13T03:41:49.912 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220756 3.53 MB/sec execute 210 sec latency 3927.193 ms 2024-04-13T03:41:50.911 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220756 3.51 MB/sec execute 211 sec latency 1810.122 ms 2024-04-13T03:41:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:50 smithi119 ceph-mon[35835]: pgmap v217: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 987 KiB/s wr, 97 op/s 2024-04-13T03:41:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:50 smithi016 ceph-mon[31772]: pgmap v217: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 987 KiB/s wr, 97 op/s 2024-04-13T03:41:51.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:50 smithi188 ceph-mon[36398]: pgmap v217: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 987 KiB/s wr, 97 op/s 2024-04-13T03:41:51.912 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220756 3.49 MB/sec execute 212 sec latency 2810.284 ms 2024-04-13T03:41:52.912 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220756 3.48 MB/sec execute 213 sec latency 3810.449 ms 2024-04-13T03:41:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:52 smithi119 ceph-mon[35835]: pgmap v218: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-13T03:41:53.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:52 smithi016 ceph-mon[31772]: pgmap v218: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-13T03:41:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:52 smithi188 ceph-mon[36398]: pgmap v218: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.3 MiB/s wr, 120 op/s 2024-04-13T03:41:53.912 INFO:tasks.workunit.client.0.smithi016.stdout: 1 220756 3.46 MB/sec execute 214 sec latency 4810.569 ms 2024-04-13T03:41:54.912 INFO:tasks.workunit.client.0.smithi016.stdout: 1 222312 3.48 MB/sec execute 215 sec latency 4824.463 ms 2024-04-13T03:41:55.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:54 smithi119 ceph-mon[35835]: pgmap v219: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 872 KiB/s wr, 80 op/s 2024-04-13T03:41:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:54 smithi016 ceph-mon[31772]: pgmap v219: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 872 KiB/s wr, 80 op/s 2024-04-13T03:41:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:54 smithi188 ceph-mon[36398]: pgmap v219: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 872 KiB/s wr, 80 op/s 2024-04-13T03:41:55.912 INFO:tasks.workunit.client.0.smithi016.stdout: 1 223340 3.49 MB/sec execute 216 sec latency 19.927 ms 2024-04-13T03:41:56.912 INFO:tasks.workunit.client.0.smithi016.stdout: 1 223969 3.50 MB/sec execute 217 sec latency 25.949 ms 2024-04-13T03:41:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:56 smithi119 ceph-mon[35835]: pgmap v220: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 872 KiB/s wr, 80 op/s 2024-04-13T03:41:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:56 smithi016 ceph-mon[31772]: pgmap v220: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 872 KiB/s wr, 80 op/s 2024-04-13T03:41:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:56 smithi188 ceph-mon[36398]: pgmap v220: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 872 KiB/s wr, 80 op/s 2024-04-13T03:41:57.913 INFO:tasks.workunit.client.0.smithi016.stdout: 1 224701 3.49 MB/sec execute 218 sec latency 19.441 ms 2024-04-13T03:41:58.913 INFO:tasks.workunit.client.0.smithi016.stdout: 1 226387 3.52 MB/sec execute 219 sec latency 17.926 ms 2024-04-13T03:41:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:41:58 smithi119 ceph-mon[35835]: pgmap v221: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-13T03:41:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:41:58 smithi016 ceph-mon[31772]: pgmap v221: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-13T03:41:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:41:58 smithi188 ceph-mon[36398]: pgmap v221: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-13T03:41:59.913 INFO:tasks.workunit.client.0.smithi016.stdout: 1 227422 3.54 MB/sec execute 220 sec latency 15.016 ms 2024-04-13T03:42:00.913 INFO:tasks.workunit.client.0.smithi016.stdout: 1 227778 3.52 MB/sec execute 221 sec latency 368.552 ms 2024-04-13T03:42:01.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:00 smithi119 ceph-mon[35835]: pgmap v222: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 786 KiB/s wr, 67 op/s 2024-04-13T03:42:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:00 smithi016 ceph-mon[31772]: pgmap v222: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 786 KiB/s wr, 67 op/s 2024-04-13T03:42:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:00 smithi188 ceph-mon[36398]: pgmap v222: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 786 KiB/s wr, 67 op/s 2024-04-13T03:42:01.913 INFO:tasks.workunit.client.0.smithi016.stdout: 1 227778 3.51 MB/sec execute 222 sec latency 1368.698 ms 2024-04-13T03:42:02.913 INFO:tasks.workunit.client.0.smithi016.stdout: 1 227778 3.49 MB/sec execute 223 sec latency 2368.862 ms 2024-04-13T03:42:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:02 smithi119 ceph-mon[35835]: pgmap v223: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-04-13T03:42:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:02 smithi119 ceph-mon[35835]: osdmap e98: 12 total, 12 up, 12 in 2024-04-13T03:42:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:02 smithi016 ceph-mon[31772]: pgmap v223: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-04-13T03:42:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:02 smithi016 ceph-mon[31772]: osdmap e98: 12 total, 12 up, 12 in 2024-04-13T03:42:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:02 smithi188 ceph-mon[36398]: pgmap v223: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1.2 MiB/s wr, 137 op/s 2024-04-13T03:42:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:02 smithi188 ceph-mon[36398]: osdmap e98: 12 total, 12 up, 12 in 2024-04-13T03:42:03.913 INFO:tasks.workunit.client.0.smithi016.stdout: 1 227778 3.48 MB/sec execute 224 sec latency 3369.070 ms 2024-04-13T03:42:04.914 INFO:tasks.workunit.client.0.smithi016.stdout: 1 229384 3.49 MB/sec execute 225 sec latency 3374.617 ms 2024-04-13T03:42:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:04 smithi119 ceph-mon[35835]: pgmap v225: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-04-13T03:42:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:04 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:42:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:04 smithi016 ceph-mon[31772]: pgmap v225: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-04-13T03:42:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:04 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:42:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:04 smithi188 ceph-mon[36398]: pgmap v225: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-04-13T03:42:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:04 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:42:05.914 INFO:tasks.workunit.client.0.smithi016.stdout: 1 230718 3.52 MB/sec execute 226 sec latency 14.246 ms 2024-04-13T03:42:06.914 INFO:tasks.workunit.client.0.smithi016.stdout: 1 231288 3.51 MB/sec execute 227 sec latency 17.469 ms 2024-04-13T03:42:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:06 smithi119 ceph-mon[35835]: pgmap v226: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-04-13T03:42:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:42:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:42:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:06 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:42:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:06 smithi016 ceph-mon[31772]: pgmap v226: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-04-13T03:42:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:42:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:42:07.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:06 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:42:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:06 smithi188 ceph-mon[36398]: pgmap v226: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-04-13T03:42:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:42:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:42:07.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:06 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:42:07.914 INFO:tasks.workunit.client.0.smithi016.stdout: 1 231395 3.50 MB/sec execute 228 sec latency 855.380 ms 2024-04-13T03:42:08.914 INFO:tasks.workunit.client.0.smithi016.stdout: 1 231395 3.49 MB/sec execute 229 sec latency 1855.523 ms 2024-04-13T03:42:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:08 smithi119 ceph-mon[35835]: pgmap v227: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 146 op/s 2024-04-13T03:42:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:08 smithi119 ceph-mon[35835]: osdmap e99: 12 total, 12 up, 12 in 2024-04-13T03:42:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:08 smithi016 ceph-mon[31772]: pgmap v227: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 146 op/s 2024-04-13T03:42:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:08 smithi016 ceph-mon[31772]: osdmap e99: 12 total, 12 up, 12 in 2024-04-13T03:42:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:08 smithi188 ceph-mon[36398]: pgmap v227: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 146 op/s 2024-04-13T03:42:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:08 smithi188 ceph-mon[36398]: osdmap e99: 12 total, 12 up, 12 in 2024-04-13T03:42:09.914 INFO:tasks.workunit.client.0.smithi016.stdout: 1 232113 3.49 MB/sec execute 230 sec latency 1857.788 ms 2024-04-13T03:42:10.914 INFO:tasks.workunit.client.0.smithi016.stdout: 1 232113 3.48 MB/sec execute 231 sec latency 1395.291 ms 2024-04-13T03:42:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:10 smithi119 ceph-mon[35835]: pgmap v229: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 952 KiB/s wr, 79 op/s 2024-04-13T03:42:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:10 smithi016 ceph-mon[31772]: pgmap v229: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 952 KiB/s wr, 79 op/s 2024-04-13T03:42:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:10 smithi188 ceph-mon[36398]: pgmap v229: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 952 KiB/s wr, 79 op/s 2024-04-13T03:42:11.915 INFO:tasks.workunit.client.0.smithi016.stdout: 1 232113 3.46 MB/sec execute 232 sec latency 2395.436 ms 2024-04-13T03:42:12.914 INFO:tasks.workunit.client.0.smithi016.stdout: 1 232113 3.45 MB/sec execute 233 sec latency 3395.525 ms 2024-04-13T03:42:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:12 smithi119 ceph-mon[35835]: pgmap v230: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 963 KiB/s wr, 95 op/s 2024-04-13T03:42:13.108 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:12 smithi016 ceph-mon[31772]: pgmap v230: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 963 KiB/s wr, 95 op/s 2024-04-13T03:42:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:12 smithi188 ceph-mon[36398]: pgmap v230: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 963 KiB/s wr, 95 op/s 2024-04-13T03:42:13.915 INFO:tasks.workunit.client.0.smithi016.stdout: 1 233059 3.44 MB/sec execute 234 sec latency 3607.467 ms 2024-04-13T03:42:14.915 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234359 3.47 MB/sec execute 235 sec latency 39.605 ms 2024-04-13T03:42:15.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:14 smithi119 ceph-mon[35835]: pgmap v231: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 947 KiB/s wr, 93 op/s 2024-04-13T03:42:15.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:14 smithi016 ceph-mon[31772]: pgmap v231: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 947 KiB/s wr, 93 op/s 2024-04-13T03:42:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:14 smithi188 ceph-mon[36398]: pgmap v231: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 947 KiB/s wr, 93 op/s 2024-04-13T03:42:15.915 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234814 3.46 MB/sec execute 236 sec latency 131.065 ms 2024-04-13T03:42:16.915 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234814 3.45 MB/sec execute 237 sec latency 1131.211 ms 2024-04-13T03:42:17.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:16 smithi016 ceph-mon[31772]: pgmap v232: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 947 KiB/s wr, 93 op/s 2024-04-13T03:42:17.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:16 smithi188 ceph-mon[36398]: pgmap v232: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 947 KiB/s wr, 93 op/s 2024-04-13T03:42:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:16 smithi119 ceph-mon[35835]: pgmap v232: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 947 KiB/s wr, 93 op/s 2024-04-13T03:42:17.915 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234814 3.43 MB/sec execute 238 sec latency 2131.299 ms 2024-04-13T03:42:18.915 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234814 3.42 MB/sec execute 239 sec latency 3131.458 ms 2024-04-13T03:42:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:18 smithi016 ceph-mon[31772]: pgmap v233: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.3 MiB/s wr, 97 op/s 2024-04-13T03:42:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:18 smithi188 ceph-mon[36398]: pgmap v233: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.3 MiB/s wr, 97 op/s 2024-04-13T03:42:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:18 smithi119 ceph-mon[35835]: pgmap v233: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.3 MiB/s wr, 97 op/s 2024-04-13T03:42:19.916 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234938 3.40 MB/sec execute 240 sec latency 3137.720 ms 2024-04-13T03:42:20.916 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234938 3.39 MB/sec execute 241 sec latency 1816.820 ms 2024-04-13T03:42:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:20 smithi016 ceph-mon[31772]: pgmap v234: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-04-13T03:42:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:20 smithi188 ceph-mon[36398]: pgmap v234: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-04-13T03:42:21.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:20 smithi119 ceph-mon[35835]: pgmap v234: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 1.2 MiB/s wr, 84 op/s 2024-04-13T03:42:21.916 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234938 3.38 MB/sec execute 242 sec latency 2816.968 ms 2024-04-13T03:42:22.916 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234938 3.36 MB/sec execute 243 sec latency 3817.062 ms 2024-04-13T03:42:23.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:22 smithi016 ceph-mon[31772]: pgmap v235: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 1.2 MiB/s wr, 95 op/s 2024-04-13T03:42:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:22 smithi188 ceph-mon[36398]: pgmap v235: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 1.2 MiB/s wr, 95 op/s 2024-04-13T03:42:23.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:22 smithi119 ceph-mon[35835]: pgmap v235: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 1.2 MiB/s wr, 95 op/s 2024-04-13T03:42:23.916 INFO:tasks.workunit.client.0.smithi016.stdout: 1 234938 3.35 MB/sec execute 244 sec latency 4817.215 ms 2024-04-13T03:42:24.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:24 smithi119 ceph-mon[35835]: pgmap v236: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 71 op/s 2024-04-13T03:42:24.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:24 smithi016 ceph-mon[31772]: pgmap v236: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 71 op/s 2024-04-13T03:42:24.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:24 smithi188 ceph-mon[36398]: pgmap v236: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 71 op/s 2024-04-13T03:42:24.916 INFO:tasks.workunit.client.0.smithi016.stdout: 1 236288 3.36 MB/sec execute 245 sec latency 4819.092 ms 2024-04-13T03:42:25.916 INFO:tasks.workunit.client.0.smithi016.stdout: 1 237657 3.39 MB/sec execute 246 sec latency 14.861 ms 2024-04-13T03:42:26.917 INFO:tasks.workunit.client.0.smithi016.stdout: 1 238254 3.38 MB/sec execute 247 sec latency 28.042 ms 2024-04-13T03:42:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:26 smithi119 ceph-mon[35835]: pgmap v237: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 71 op/s 2024-04-13T03:42:27.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:26 smithi016 ceph-mon[31772]: pgmap v237: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 71 op/s 2024-04-13T03:42:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:26 smithi188 ceph-mon[36398]: pgmap v237: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 1.1 MiB/s wr, 71 op/s 2024-04-13T03:42:27.917 INFO:tasks.workunit.client.0.smithi016.stdout: 1 238487 3.37 MB/sec execute 248 sec latency 706.348 ms 2024-04-13T03:42:28.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:27 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:42:28.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:27 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:42:28.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:27 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:42:28.917 INFO:tasks.workunit.client.0.smithi016.stdout: 1 238487 3.36 MB/sec execute 249 sec latency 1706.584 ms 2024-04-13T03:42:29.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:28 smithi119 ceph-mon[35835]: pgmap v238: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 1.3 MiB/s wr, 107 op/s 2024-04-13T03:42:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:28 smithi016 ceph-mon[31772]: pgmap v238: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 1.3 MiB/s wr, 107 op/s 2024-04-13T03:42:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:28 smithi188 ceph-mon[36398]: pgmap v238: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 1.3 MiB/s wr, 107 op/s 2024-04-13T03:42:29.917 INFO:tasks.workunit.client.0.smithi016.stdout: 1 239837 3.37 MB/sec execute 250 sec latency 1739.667 ms 2024-04-13T03:42:30.917 INFO:tasks.workunit.client.0.smithi016.stdout: 1 241236 3.40 MB/sec execute 251 sec latency 19.338 ms 2024-04-13T03:42:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:30 smithi119 ceph-mon[35835]: pgmap v239: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 370 KiB/s wr, 50 op/s 2024-04-13T03:42:31.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:30 smithi016 ceph-mon[31772]: pgmap v239: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 370 KiB/s wr, 50 op/s 2024-04-13T03:42:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:30 smithi188 ceph-mon[36398]: pgmap v239: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 370 KiB/s wr, 50 op/s 2024-04-13T03:42:31.917 INFO:tasks.workunit.client.0.smithi016.stdout: 1 241836 3.39 MB/sec execute 252 sec latency 19.198 ms 2024-04-13T03:42:32.917 INFO:tasks.workunit.client.0.smithi016.stdout: 1 241961 3.38 MB/sec execute 253 sec latency 859.925 ms 2024-04-13T03:42:33.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:32 smithi119 ceph-mon[35835]: pgmap v240: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 994 KiB/s wr, 113 op/s 2024-04-13T03:42:33.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:32 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:42:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:32 smithi016 ceph-mon[31772]: pgmap v240: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 994 KiB/s wr, 113 op/s 2024-04-13T03:42:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:32 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:42:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:32 smithi188 ceph-mon[36398]: pgmap v240: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 994 KiB/s wr, 113 op/s 2024-04-13T03:42:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:32 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:42:33.917 INFO:tasks.workunit.client.0.smithi016.stdout: 1 241962 3.37 MB/sec execute 254 sec latency 1858.541 ms 2024-04-13T03:42:34.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:34 smithi119 ceph-mon[35835]: pgmap v241: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 890 KiB/s wr, 98 op/s 2024-04-13T03:42:34.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:34 smithi016 ceph-mon[31772]: pgmap v241: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 890 KiB/s wr, 98 op/s 2024-04-13T03:42:34.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:34 smithi188 ceph-mon[36398]: pgmap v241: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 890 KiB/s wr, 98 op/s 2024-04-13T03:42:34.918 INFO:tasks.workunit.client.0.smithi016.stdout: 1 242029 3.35 MB/sec execute 255 sec latency 900.326 ms 2024-04-13T03:42:35.918 INFO:tasks.workunit.client.0.smithi016.stdout: 1 242029 3.34 MB/sec execute 256 sec latency 1900.454 ms 2024-04-13T03:42:36.918 INFO:tasks.workunit.client.0.smithi016.stdout: 1 242029 3.33 MB/sec execute 257 sec latency 2900.555 ms 2024-04-13T03:42:37.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:36 smithi119 ceph-mon[35835]: pgmap v242: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 890 KiB/s wr, 98 op/s 2024-04-13T03:42:37.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:36 smithi016 ceph-mon[31772]: pgmap v242: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 890 KiB/s wr, 98 op/s 2024-04-13T03:42:37.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:36 smithi188 ceph-mon[36398]: pgmap v242: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 890 KiB/s wr, 98 op/s 2024-04-13T03:42:37.918 INFO:tasks.workunit.client.0.smithi016.stdout: 1 242029 3.32 MB/sec execute 258 sec latency 3900.721 ms 2024-04-13T03:42:38.918 INFO:tasks.workunit.client.0.smithi016.stdout: 1 242030 3.30 MB/sec execute 259 sec latency 4898.535 ms 2024-04-13T03:42:39.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:38 smithi119 ceph-mon[35835]: pgmap v243: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 989 KiB/s wr, 118 op/s 2024-04-13T03:42:39.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:38 smithi016 ceph-mon[31772]: pgmap v243: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 989 KiB/s wr, 118 op/s 2024-04-13T03:42:39.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:38 smithi188 ceph-mon[36398]: pgmap v243: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 989 KiB/s wr, 118 op/s 2024-04-13T03:42:39.918 INFO:tasks.workunit.client.0.smithi016.stdout: 1 242641 3.31 MB/sec execute 260 sec latency 405.372 ms 2024-04-13T03:42:40.918 INFO:tasks.workunit.client.0.smithi016.stdout: 1 242641 3.30 MB/sec execute 261 sec latency 1405.545 ms 2024-04-13T03:42:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:40 smithi119 ceph-mon[35835]: pgmap v244: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 723 KiB/s wr, 82 op/s 2024-04-13T03:42:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:40 smithi016 ceph-mon[31772]: pgmap v244: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 723 KiB/s wr, 82 op/s 2024-04-13T03:42:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:40 smithi188 ceph-mon[36398]: pgmap v244: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 723 KiB/s wr, 82 op/s 2024-04-13T03:42:41.919 INFO:tasks.workunit.client.0.smithi016.stdout: 1 242808 3.28 MB/sec execute 262 sec latency 2243.135 ms 2024-04-13T03:42:42.919 INFO:tasks.workunit.client.0.smithi016.stdout: 1 244226 3.30 MB/sec execute 263 sec latency 25.591 ms 2024-04-13T03:42:43.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:42 smithi119 ceph-mon[35835]: pgmap v245: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-13T03:42:43.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:42 smithi016 ceph-mon[31772]: pgmap v245: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-13T03:42:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:42 smithi188 ceph-mon[36398]: pgmap v245: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 1.1 MiB/s wr, 108 op/s 2024-04-13T03:42:43.919 INFO:tasks.workunit.client.0.smithi016.stdout: 1 245186 3.31 MB/sec execute 264 sec latency 45.705 ms 2024-04-13T03:42:44.919 INFO:tasks.workunit.client.0.smithi016.stdout: 1 245922 3.31 MB/sec execute 265 sec latency 17.253 ms 2024-04-13T03:42:45.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:44 smithi119 ceph-mon[35835]: pgmap v246: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 518 KiB/s wr, 46 op/s 2024-04-13T03:42:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:44 smithi016 ceph-mon[31772]: pgmap v246: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 518 KiB/s wr, 46 op/s 2024-04-13T03:42:45.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:44 smithi188 ceph-mon[36398]: pgmap v246: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 518 KiB/s wr, 46 op/s 2024-04-13T03:42:45.919 INFO:tasks.workunit.client.0.smithi016.stdout: 1 247228 3.31 MB/sec execute 266 sec latency 18.487 ms 2024-04-13T03:42:46.919 INFO:tasks.workunit.client.0.smithi016.stdout: 1 248476 3.34 MB/sec execute 267 sec latency 16.384 ms 2024-04-13T03:42:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:46 smithi119 ceph-mon[35835]: pgmap v247: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 518 KiB/s wr, 46 op/s 2024-04-13T03:42:47.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:46 smithi016 ceph-mon[31772]: pgmap v247: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 518 KiB/s wr, 46 op/s 2024-04-13T03:42:47.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:46 smithi188 ceph-mon[36398]: pgmap v247: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 518 KiB/s wr, 46 op/s 2024-04-13T03:42:47.919 INFO:tasks.workunit.client.0.smithi016.stdout: 1 248927 3.33 MB/sec execute 268 sec latency 30.198 ms 2024-04-13T03:42:48.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 249908 3.34 MB/sec execute 269 sec latency 23.467 ms 2024-04-13T03:42:49.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:48 smithi119 ceph-mon[35835]: pgmap v248: 129 pgs: 129 active+clean; 174 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-04-13T03:42:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:48 smithi016 ceph-mon[31772]: pgmap v248: 129 pgs: 129 active+clean; 174 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-04-13T03:42:49.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:48 smithi188 ceph-mon[36398]: pgmap v248: 129 pgs: 129 active+clean; 174 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-04-13T03:42:49.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 251260 3.35 MB/sec execute 270 sec latency 21.022 ms 2024-04-13T03:42:50.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 252203 3.37 MB/sec execute 271 sec latency 18.210 ms 2024-04-13T03:42:51.074 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:50 smithi119 ceph-mon[35835]: pgmap v249: 129 pgs: 129 active+clean; 174 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1003 KiB/s wr, 96 op/s 2024-04-13T03:42:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:50 smithi016 ceph-mon[31772]: pgmap v249: 129 pgs: 129 active+clean; 174 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1003 KiB/s wr, 96 op/s 2024-04-13T03:42:51.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:50 smithi188 ceph-mon[36398]: pgmap v249: 129 pgs: 129 active+clean; 174 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1003 KiB/s wr, 96 op/s 2024-04-13T03:42:51.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 252816 3.36 MB/sec execute 272 sec latency 27.580 ms 2024-04-13T03:42:52.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 253796 3.36 MB/sec execute 273 sec latency 20.036 ms 2024-04-13T03:42:53.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:52 smithi119 ceph-mon[35835]: pgmap v250: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-13T03:42:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:52 smithi016 ceph-mon[31772]: pgmap v250: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-13T03:42:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:52 smithi188 ceph-mon[36398]: pgmap v250: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-13T03:42:53.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 255214 3.38 MB/sec execute 274 sec latency 19.176 ms 2024-04-13T03:42:54.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 255859 3.39 MB/sec execute 275 sec latency 47.857 ms 2024-04-13T03:42:55.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:54 smithi119 ceph-mon[35835]: pgmap v251: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-13T03:42:55.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:54 smithi016 ceph-mon[31772]: pgmap v251: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-13T03:42:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:54 smithi188 ceph-mon[36398]: pgmap v251: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-13T03:42:55.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 256608 3.38 MB/sec execute 276 sec latency 24.623 ms 2024-04-13T03:42:56.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 257870 3.39 MB/sec execute 277 sec latency 24.506 ms 2024-04-13T03:42:57.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:56 smithi119 ceph-mon[35835]: pgmap v252: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-13T03:42:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:56 smithi016 ceph-mon[31772]: pgmap v252: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-13T03:42:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:56 smithi188 ceph-mon[36398]: pgmap v252: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-13T03:42:57.920 INFO:tasks.workunit.client.0.smithi016.stdout: 1 259106 3.41 MB/sec execute 278 sec latency 20.423 ms 2024-04-13T03:42:58.921 INFO:tasks.workunit.client.0.smithi016.stdout: 1 259565 3.41 MB/sec execute 279 sec latency 26.094 ms 2024-04-13T03:42:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:42:58 smithi119 ceph-mon[35835]: pgmap v253: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-04-13T03:42:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:42:58 smithi016 ceph-mon[31772]: pgmap v253: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-04-13T03:42:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:42:58 smithi188 ceph-mon[36398]: pgmap v253: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 200 op/s 2024-04-13T03:42:59.921 INFO:tasks.workunit.client.0.smithi016.stdout: 1 260540 3.41 MB/sec execute 280 sec latency 27.192 ms 2024-04-13T03:43:00.921 INFO:tasks.workunit.client.0.smithi016.stdout: 1 261957 3.43 MB/sec execute 281 sec latency 19.599 ms 2024-04-13T03:43:01.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:00 smithi119 ceph-mon[35835]: pgmap v254: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-04-13T03:43:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:00 smithi016 ceph-mon[31772]: pgmap v254: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-04-13T03:43:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:00 smithi188 ceph-mon[36398]: pgmap v254: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-04-13T03:43:01.921 INFO:tasks.workunit.client.0.smithi016.stdout: 1 262227 3.42 MB/sec execute 282 sec latency 103.420 ms 2024-04-13T03:43:02.921 INFO:tasks.workunit.client.0.smithi016.stdout: 1 262907 3.43 MB/sec execute 283 sec latency 68.255 ms 2024-04-13T03:43:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:02 smithi119 ceph-mon[35835]: pgmap v255: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 225 op/s 2024-04-13T03:43:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:02 smithi119 ceph-mon[35835]: osdmap e100: 12 total, 12 up, 12 in 2024-04-13T03:43:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:02 smithi016 ceph-mon[31772]: pgmap v255: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 225 op/s 2024-04-13T03:43:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:02 smithi016 ceph-mon[31772]: osdmap e100: 12 total, 12 up, 12 in 2024-04-13T03:43:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:02 smithi188 ceph-mon[36398]: pgmap v255: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 MiB/s wr, 225 op/s 2024-04-13T03:43:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:02 smithi188 ceph-mon[36398]: osdmap e100: 12 total, 12 up, 12 in 2024-04-13T03:43:03.921 INFO:tasks.workunit.client.0.smithi016.stdout: 1 263606 3.42 MB/sec execute 284 sec latency 21.873 ms 2024-04-13T03:43:04.921 INFO:tasks.workunit.client.0.smithi016.stdout: 1 264654 3.42 MB/sec execute 285 sec latency 23.325 ms 2024-04-13T03:43:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:04 smithi119 ceph-mon[35835]: pgmap v257: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-13T03:43:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:04 smithi016 ceph-mon[31772]: pgmap v257: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-13T03:43:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:04 smithi188 ceph-mon[36398]: pgmap v257: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-13T03:43:05.921 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266064 3.45 MB/sec execute 286 sec latency 23.973 ms 2024-04-13T03:43:06.921 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266617 3.45 MB/sec execute 287 sec latency 23.472 ms 2024-04-13T03:43:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:06 smithi119 ceph-mon[35835]: pgmap v258: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-13T03:43:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:43:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:06 smithi016 ceph-mon[31772]: pgmap v258: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-13T03:43:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:43:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:06 smithi188 ceph-mon[36398]: pgmap v258: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-13T03:43:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:43:07.922 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266654 3.43 MB/sec execute 288 sec latency 942.358 ms 2024-04-13T03:43:08.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:07 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:43:08.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:07 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:43:08.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:07 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:43:08.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:07 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:43:08.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:07 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:43:08.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:07 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:43:08.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:07 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:43:08.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:07 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:43:08.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:07 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:43:08.922 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266654 3.42 MB/sec execute 289 sec latency 1942.451 ms 2024-04-13T03:43:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:08 smithi119 ceph-mon[35835]: pgmap v259: 129 pgs: 129 active+clean; 190 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-04-13T03:43:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:08 smithi119 ceph-mon[35835]: osdmap e101: 12 total, 12 up, 12 in 2024-04-13T03:43:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:08 smithi016 ceph-mon[31772]: pgmap v259: 129 pgs: 129 active+clean; 190 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-04-13T03:43:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:08 smithi016 ceph-mon[31772]: osdmap e101: 12 total, 12 up, 12 in 2024-04-13T03:43:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:08 smithi188 ceph-mon[36398]: pgmap v259: 129 pgs: 129 active+clean; 190 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-04-13T03:43:09.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:08 smithi188 ceph-mon[36398]: osdmap e101: 12 total, 12 up, 12 in 2024-04-13T03:43:09.922 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266654 3.41 MB/sec execute 290 sec latency 2942.574 ms 2024-04-13T03:43:10.922 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266654 3.40 MB/sec execute 291 sec latency 3942.743 ms 2024-04-13T03:43:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:10 smithi119 ceph-mon[35835]: pgmap v261: 129 pgs: 129 active+clean; 190 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-04-13T03:43:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:10 smithi016 ceph-mon[31772]: pgmap v261: 129 pgs: 129 active+clean; 190 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-04-13T03:43:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:10 smithi188 ceph-mon[36398]: pgmap v261: 129 pgs: 129 active+clean; 190 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.1 MiB/s wr, 116 op/s 2024-04-13T03:43:11.922 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266654 3.39 MB/sec execute 292 sec latency 4942.851 ms 2024-04-13T03:43:12.922 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266654 3.37 MB/sec execute 293 sec latency 5942.983 ms 2024-04-13T03:43:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:12 smithi119 ceph-mon[35835]: pgmap v262: 129 pgs: 129 active+clean; 192 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-04-13T03:43:13.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:12 smithi016 ceph-mon[31772]: pgmap v262: 129 pgs: 129 active+clean; 192 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-04-13T03:43:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:12 smithi188 ceph-mon[36398]: pgmap v262: 129 pgs: 129 active+clean; 192 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 1.0 MiB/s wr, 112 op/s 2024-04-13T03:43:13.923 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266654 3.36 MB/sec execute 294 sec latency 6943.144 ms 2024-04-13T03:43:14.922 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266654 3.35 MB/sec execute 295 sec latency 7943.249 ms 2024-04-13T03:43:15.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:14 smithi119 ceph-mon[35835]: pgmap v263: 129 pgs: 129 active+clean; 192 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-13T03:43:15.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:14 smithi016 ceph-mon[31772]: pgmap v263: 129 pgs: 129 active+clean; 192 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-13T03:43:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:14 smithi188 ceph-mon[36398]: pgmap v263: 129 pgs: 129 active+clean; 192 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-13T03:43:15.923 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266654 3.34 MB/sec execute 296 sec latency 8943.351 ms 2024-04-13T03:43:16.923 INFO:tasks.workunit.client.0.smithi016.stdout: 1 266793 3.33 MB/sec execute 297 sec latency 9801.531 ms 2024-04-13T03:43:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:16 smithi119 ceph-mon[35835]: pgmap v264: 129 pgs: 129 active+clean; 192 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-13T03:43:17.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:16 smithi016 ceph-mon[31772]: pgmap v264: 129 pgs: 129 active+clean; 192 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-13T03:43:17.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:16 smithi188 ceph-mon[36398]: pgmap v264: 129 pgs: 129 active+clean; 192 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 1.0 MiB/s wr, 110 op/s 2024-04-13T03:43:17.923 INFO:tasks.workunit.client.0.smithi016.stdout: 1 267752 3.34 MB/sec execute 298 sec latency 24.086 ms 2024-04-13T03:43:18.923 INFO:tasks.workunit.client.0.smithi016.stdout: 1 268975 3.35 MB/sec execute 299 sec latency 216.728 ms 2024-04-13T03:43:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:18 smithi119 ceph-mon[35835]: pgmap v265: 129 pgs: 129 active+clean; 195 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 505 KiB/s wr, 34 op/s 2024-04-13T03:43:19.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:18 smithi016 ceph-mon[31772]: pgmap v265: 129 pgs: 129 active+clean; 195 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 505 KiB/s wr, 34 op/s 2024-04-13T03:43:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:18 smithi188 ceph-mon[36398]: pgmap v265: 129 pgs: 129 active+clean; 195 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 505 KiB/s wr, 34 op/s 2024-04-13T03:43:19.923 INFO:tasks.workunit.client.0.smithi016.stdout: 1 269972 3.36 MB/sec execute 300 sec latency 17.240 ms 2024-04-13T03:43:20.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 270745 3.36 MB/sec execute 301 sec latency 35.630 ms 2024-04-13T03:43:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:20 smithi119 ceph-mon[35835]: pgmap v266: 129 pgs: 129 active+clean; 195 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 436 KiB/s wr, 30 op/s 2024-04-13T03:43:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:20 smithi016 ceph-mon[31772]: pgmap v266: 129 pgs: 129 active+clean; 195 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 436 KiB/s wr, 30 op/s 2024-04-13T03:43:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:20 smithi188 ceph-mon[36398]: pgmap v266: 129 pgs: 129 active+clean; 195 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 436 KiB/s wr, 30 op/s 2024-04-13T03:43:21.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 271497 3.36 MB/sec execute 302 sec latency 254.193 ms 2024-04-13T03:43:22.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 272797 3.37 MB/sec execute 303 sec latency 22.979 ms 2024-04-13T03:43:23.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:22 smithi016 ceph-mon[31772]: pgmap v267: 129 pgs: 129 active+clean; 197 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-13T03:43:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:22 smithi188 ceph-mon[36398]: pgmap v267: 129 pgs: 129 active+clean; 197 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-13T03:43:23.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:22 smithi119 ceph-mon[35835]: pgmap v267: 129 pgs: 129 active+clean; 197 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-13T03:43:23.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 273530 3.38 MB/sec execute 304 sec latency 20.258 ms 2024-04-13T03:43:24.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 274132 3.37 MB/sec execute 305 sec latency 23.565 ms 2024-04-13T03:43:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:24 smithi016 ceph-mon[31772]: pgmap v268: 129 pgs: 129 active+clean; 197 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-13T03:43:25.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:24 smithi188 ceph-mon[36398]: pgmap v268: 129 pgs: 129 active+clean; 197 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-13T03:43:25.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:24 smithi119 ceph-mon[35835]: pgmap v268: 129 pgs: 129 active+clean; 197 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-13T03:43:25.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 274843 3.37 MB/sec execute 306 sec latency 260.220 ms 2024-04-13T03:43:26.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 275042 3.36 MB/sec execute 307 sec latency 1105.158 ms 2024-04-13T03:43:27.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:26 smithi016 ceph-mon[31772]: pgmap v269: 129 pgs: 129 active+clean; 197 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-13T03:43:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:26 smithi188 ceph-mon[36398]: pgmap v269: 129 pgs: 129 active+clean; 197 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-13T03:43:27.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:26 smithi119 ceph-mon[35835]: pgmap v269: 129 pgs: 129 active+clean; 197 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-13T03:43:27.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 276326 3.38 MB/sec execute 308 sec latency 23.981 ms 2024-04-13T03:43:28.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277076 3.39 MB/sec execute 309 sec latency 18.363 ms 2024-04-13T03:43:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:28 smithi016 ceph-mon[31772]: pgmap v270: 129 pgs: 129 active+clean; 200 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 164 op/s 2024-04-13T03:43:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:28 smithi188 ceph-mon[36398]: pgmap v270: 129 pgs: 129 active+clean; 200 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 164 op/s 2024-04-13T03:43:29.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:28 smithi119 ceph-mon[35835]: pgmap v270: 129 pgs: 129 active+clean; 200 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 164 op/s 2024-04-13T03:43:29.924 INFO:tasks.workunit.client.0.smithi016.stdout: 1 277697 3.38 MB/sec execute 310 sec latency 22.644 ms 2024-04-13T03:43:30.925 INFO:tasks.workunit.client.0.smithi016.stdout: 1 278385 3.38 MB/sec execute 311 sec latency 291.588 ms 2024-04-13T03:43:31.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:30 smithi016 ceph-mon[31772]: pgmap v271: 129 pgs: 129 active+clean; 200 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 149 op/s 2024-04-13T03:43:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:30 smithi188 ceph-mon[36398]: pgmap v271: 129 pgs: 129 active+clean; 200 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 149 op/s 2024-04-13T03:43:31.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:30 smithi119 ceph-mon[35835]: pgmap v271: 129 pgs: 129 active+clean; 200 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 149 op/s 2024-04-13T03:43:31.925 INFO:tasks.workunit.client.0.smithi016.stdout: 1 278608 3.37 MB/sec execute 312 sec latency 1132.224 ms 2024-04-13T03:43:32.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 280056 3.39 MB/sec execute 313 sec latency 19.146 ms 2024-04-13T03:43:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:32 smithi016 ceph-mon[31772]: pgmap v272: 129 pgs: 129 active+clean; 203 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 211 op/s 2024-04-13T03:43:33.122 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:32 smithi119 ceph-mon[35835]: pgmap v272: 129 pgs: 129 active+clean; 203 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 211 op/s 2024-04-13T03:43:33.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:32 smithi188 ceph-mon[36398]: pgmap v272: 129 pgs: 129 active+clean; 203 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 211 op/s 2024-04-13T03:43:33.925 INFO:tasks.workunit.client.0.smithi016.stdout: 1 280710 3.39 MB/sec execute 314 sec latency 15.799 ms 2024-04-13T03:43:34.925 INFO:tasks.workunit.client.0.smithi016.stdout: 1 281633 3.40 MB/sec execute 315 sec latency 23.620 ms 2024-04-13T03:43:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:34 smithi188 ceph-mon[36398]: pgmap v273: 129 pgs: 129 active+clean; 203 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-13T03:43:35.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:34 smithi119 ceph-mon[35835]: pgmap v273: 129 pgs: 129 active+clean; 203 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-13T03:43:35.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:34 smithi016 ceph-mon[31772]: pgmap v273: 129 pgs: 129 active+clean; 203 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-13T03:43:35.925 INFO:tasks.workunit.client.0.smithi016.stdout: 1 281934 3.39 MB/sec execute 316 sec latency 365.791 ms 2024-04-13T03:43:36.925 INFO:tasks.workunit.client.0.smithi016.stdout: 1 282157 3.38 MB/sec execute 317 sec latency 1206.665 ms 2024-04-13T03:43:37.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:36 smithi188 ceph-mon[36398]: pgmap v274: 129 pgs: 129 active+clean; 203 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-13T03:43:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:36 smithi119 ceph-mon[35835]: pgmap v274: 129 pgs: 129 active+clean; 203 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-13T03:43:37.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:36 smithi016 ceph-mon[31772]: pgmap v274: 129 pgs: 129 active+clean; 203 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-13T03:43:37.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 283615 3.40 MB/sec execute 318 sec latency 20.192 ms 2024-04-13T03:43:38.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 284249 3.40 MB/sec execute 319 sec latency 20.617 ms 2024-04-13T03:43:39.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:38 smithi188 ceph-mon[36398]: pgmap v275: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 185 op/s 2024-04-13T03:43:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:38 smithi119 ceph-mon[35835]: pgmap v275: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 185 op/s 2024-04-13T03:43:39.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:38 smithi016 ceph-mon[31772]: pgmap v275: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 185 op/s 2024-04-13T03:43:39.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 285035 3.40 MB/sec execute 320 sec latency 18.171 ms 2024-04-13T03:43:40.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 285476 3.39 MB/sec execute 321 sec latency 579.058 ms 2024-04-13T03:43:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:40 smithi188 ceph-mon[36398]: pgmap v276: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:43:41.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:40 smithi119 ceph-mon[35835]: pgmap v276: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:43:41.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:40 smithi016 ceph-mon[31772]: pgmap v276: 129 pgs: 129 active+clean; 209 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:43:41.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 285678 3.38 MB/sec execute 322 sec latency 1418.419 ms 2024-04-13T03:43:42.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 286994 3.40 MB/sec execute 323 sec latency 20.813 ms 2024-04-13T03:43:43.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:42 smithi016 ceph-mon[31772]: pgmap v277: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-04-13T03:43:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:42 smithi119 ceph-mon[35835]: pgmap v277: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-04-13T03:43:43.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:42 smithi188 ceph-mon[36398]: pgmap v277: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-04-13T03:43:43.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 287768 3.41 MB/sec execute 324 sec latency 18.874 ms 2024-04-13T03:43:44.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 288502 3.40 MB/sec execute 325 sec latency 18.913 ms 2024-04-13T03:43:45.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:44 smithi119 ceph-mon[35835]: pgmap v278: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-13T03:43:45.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:44 smithi016 ceph-mon[31772]: pgmap v278: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-13T03:43:45.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:44 smithi188 ceph-mon[36398]: pgmap v278: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-13T03:43:45.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 289024 3.40 MB/sec execute 326 sec latency 465.729 ms 2024-04-13T03:43:46.926 INFO:tasks.workunit.client.0.smithi016.stdout: 1 289247 3.39 MB/sec execute 327 sec latency 1300.161 ms 2024-04-13T03:43:47.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:46 smithi119 ceph-mon[35835]: pgmap v279: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-13T03:43:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:46 smithi016 ceph-mon[31772]: pgmap v279: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-13T03:43:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:46 smithi188 ceph-mon[36398]: pgmap v279: 129 pgs: 129 active+clean; 212 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-13T03:43:47.927 INFO:tasks.workunit.client.0.smithi016.stdout: 1 290605 3.41 MB/sec execute 328 sec latency 20.815 ms 2024-04-13T03:43:48.927 INFO:tasks.workunit.client.0.smithi016.stdout: 1 291325 3.41 MB/sec execute 329 sec latency 19.886 ms 2024-04-13T03:43:49.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:48 smithi119 ceph-mon[35835]: pgmap v280: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 168 op/s 2024-04-13T03:43:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:48 smithi016 ceph-mon[31772]: pgmap v280: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 168 op/s 2024-04-13T03:43:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:48 smithi188 ceph-mon[36398]: pgmap v280: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.6 MiB/s wr, 168 op/s 2024-04-13T03:43:49.927 INFO:tasks.workunit.client.0.smithi016.stdout: 1 292083 3.41 MB/sec execute 330 sec latency 18.297 ms 2024-04-13T03:43:50.927 INFO:tasks.workunit.client.0.smithi016.stdout: 1 292567 3.41 MB/sec execute 331 sec latency 515.366 ms 2024-04-13T03:43:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:50 smithi119 ceph-mon[35835]: pgmap v281: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 908 KiB/s wr, 108 op/s 2024-04-13T03:43:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:50 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:43:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:50 smithi016 ceph-mon[31772]: pgmap v281: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 908 KiB/s wr, 108 op/s 2024-04-13T03:43:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:50 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:43:51.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:50 smithi188 ceph-mon[36398]: pgmap v281: 129 pgs: 129 active+clean; 215 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 908 KiB/s wr, 108 op/s 2024-04-13T03:43:51.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:50 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:43:51.927 INFO:tasks.workunit.client.0.smithi016.stdout: 1 292790 3.40 MB/sec execute 332 sec latency 1350.090 ms 2024-04-13T03:43:52.927 INFO:tasks.workunit.client.0.smithi016.stdout: 1 294097 3.41 MB/sec execute 333 sec latency 21.957 ms 2024-04-13T03:43:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:52 smithi119 ceph-mon[35835]: pgmap v282: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 166 op/s 2024-04-13T03:43:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:52 smithi016 ceph-mon[31772]: pgmap v282: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 166 op/s 2024-04-13T03:43:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:52 smithi188 ceph-mon[36398]: pgmap v282: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 166 op/s 2024-04-13T03:43:53.927 INFO:tasks.workunit.client.0.smithi016.stdout: 1 294830 3.42 MB/sec execute 334 sec latency 133.079 ms 2024-04-13T03:43:54.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:53 smithi188 ceph-mon[36398]: pgmap v283: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:43:54.274 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:53 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:43:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:53 smithi119 ceph-mon[35835]: pgmap v283: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:43:54.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:53 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:43:54.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:53 smithi016 ceph-mon[31772]: pgmap v283: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:43:54.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:53 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:43:54.927 INFO:tasks.workunit.client.0.smithi016.stdout: 1 295588 3.41 MB/sec execute 335 sec latency 27.422 ms 2024-04-13T03:43:55.928 INFO:tasks.workunit.client.0.smithi016.stdout: 1 296117 3.41 MB/sec execute 336 sec latency 447.859 ms 2024-04-13T03:43:56.928 INFO:tasks.workunit.client.0.smithi016.stdout: 1 296340 3.40 MB/sec execute 337 sec latency 1280.787 ms 2024-04-13T03:43:57.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:56 smithi119 ceph-mon[35835]: pgmap v284: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:43:57.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:56 smithi016 ceph-mon[31772]: pgmap v284: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:43:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:56 smithi188 ceph-mon[36398]: pgmap v284: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:43:57.928 INFO:tasks.workunit.client.0.smithi016.stdout: 1 297818 3.42 MB/sec execute 338 sec latency 20.597 ms 2024-04-13T03:43:58.928 INFO:tasks.workunit.client.0.smithi016.stdout: 1 298630 3.42 MB/sec execute 339 sec latency 26.445 ms 2024-04-13T03:43:59.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:43:58 smithi119 ceph-mon[35835]: pgmap v285: 129 pgs: 129 active+clean; 224 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-13T03:43:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:43:58 smithi016 ceph-mon[31772]: pgmap v285: 129 pgs: 129 active+clean; 224 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-13T03:43:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:43:58 smithi188 ceph-mon[36398]: pgmap v285: 129 pgs: 129 active+clean; 224 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-13T03:43:59.928 INFO:tasks.workunit.client.0.smithi016.stdout: 1 299481 3.43 MB/sec execute 340 sec latency 26.030 ms 2024-04-13T03:44:00.928 INFO:tasks.workunit.client.0.smithi016.stdout: 1 299665 3.42 MB/sec execute 341 sec latency 725.606 ms 2024-04-13T03:44:01.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:00 smithi119 ceph-mon[35835]: pgmap v286: 129 pgs: 129 active+clean; 224 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-13T03:44:01.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:00 smithi016 ceph-mon[31772]: pgmap v286: 129 pgs: 129 active+clean; 224 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-13T03:44:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:00 smithi188 ceph-mon[36398]: pgmap v286: 129 pgs: 129 active+clean; 224 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-04-13T03:44:01.929 INFO:tasks.workunit.client.0.smithi016.stdout: 1 299867 3.41 MB/sec execute 342 sec latency 1560.952 ms 2024-04-13T03:44:02.929 INFO:tasks.workunit.client.0.smithi016.stdout: 1 300262 3.40 MB/sec execute 343 sec latency 91.720 ms 2024-04-13T03:44:03.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:02 smithi119 ceph-mon[35835]: pgmap v287: 129 pgs: 129 active+clean; 227 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 184 op/s 2024-04-13T03:44:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:02 smithi016 ceph-mon[31772]: pgmap v287: 129 pgs: 129 active+clean; 227 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 184 op/s 2024-04-13T03:44:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:02 smithi188 ceph-mon[36398]: pgmap v287: 129 pgs: 129 active+clean; 227 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 184 op/s 2024-04-13T03:44:03.929 INFO:tasks.workunit.client.0.smithi016.stdout: 1 301161 3.41 MB/sec execute 344 sec latency 97.616 ms 2024-04-13T03:44:04.929 INFO:tasks.workunit.client.0.smithi016.stdout: 1 302001 3.42 MB/sec execute 345 sec latency 15.044 ms 2024-04-13T03:44:05.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:04 smithi119 ceph-mon[35835]: pgmap v288: 129 pgs: 129 active+clean; 227 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-13T03:44:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:04 smithi016 ceph-mon[31772]: pgmap v288: 129 pgs: 129 active+clean; 227 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-13T03:44:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:04 smithi188 ceph-mon[36398]: pgmap v288: 129 pgs: 129 active+clean; 227 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-13T03:44:05.929 INFO:tasks.workunit.client.0.smithi016.stdout: 1 302947 3.43 MB/sec execute 346 sec latency 37.848 ms 2024-04-13T03:44:06.929 INFO:tasks.workunit.client.0.smithi016.stdout: 1 303501 3.42 MB/sec execute 347 sec latency 468.907 ms 2024-04-13T03:44:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:06 smithi119 ceph-mon[35835]: pgmap v289: 129 pgs: 129 active+clean; 227 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-13T03:44:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:06 smithi016 ceph-mon[31772]: pgmap v289: 129 pgs: 129 active+clean; 227 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-13T03:44:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:06 smithi188 ceph-mon[36398]: pgmap v289: 129 pgs: 129 active+clean; 227 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 126 op/s 2024-04-13T03:44:07.929 INFO:tasks.workunit.client.0.smithi016.stdout: 1 304729 3.43 MB/sec execute 348 sec latency 20.857 ms 2024-04-13T03:44:08.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:07 smithi119 ceph-mon[35835]: osdmap e102: 12 total, 12 up, 12 in 2024-04-13T03:44:08.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:07 smithi016 ceph-mon[31772]: osdmap e102: 12 total, 12 up, 12 in 2024-04-13T03:44:08.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:07 smithi188 ceph-mon[36398]: osdmap e102: 12 total, 12 up, 12 in 2024-04-13T03:44:08.929 INFO:tasks.workunit.client.0.smithi016.stdout: 1 305514 3.44 MB/sec execute 349 sec latency 18.447 ms 2024-04-13T03:44:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:08 smithi119 ceph-mon[35835]: pgmap v291: 129 pgs: 129 active+clean; 231 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-04-13T03:44:09.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:08 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:44:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:08 smithi016 ceph-mon[31772]: pgmap v291: 129 pgs: 129 active+clean; 231 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-04-13T03:44:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:08 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:44:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:08 smithi188 ceph-mon[36398]: pgmap v291: 129 pgs: 129 active+clean; 231 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-04-13T03:44:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:08 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:44:09.929 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306453 3.44 MB/sec execute 350 sec latency 21.770 ms 2024-04-13T03:44:10.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:09 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:44:10.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:09 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:44:10.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:09 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:44:10.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:09 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:44:10.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:09 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:44:10.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:09 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:44:10.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:09 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:44:10.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:09 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:44:10.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:09 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:44:10.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306757 3.43 MB/sec execute 351 sec latency 411.702 ms 2024-04-13T03:44:11.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:10 smithi119 ceph-mon[35835]: pgmap v292: 129 pgs: 129 active+clean; 231 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-04-13T03:44:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:10 smithi016 ceph-mon[31772]: pgmap v292: 129 pgs: 129 active+clean; 231 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-04-13T03:44:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:10 smithi188 ceph-mon[36398]: pgmap v292: 129 pgs: 129 active+clean; 231 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-04-13T03:44:11.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306959 3.42 MB/sec execute 352 sec latency 1241.723 ms 2024-04-13T03:44:12.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306983 3.41 MB/sec execute 353 sec latency 978.326 ms 2024-04-13T03:44:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:12 smithi119 ceph-mon[35835]: pgmap v293: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-13T03:44:13.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:12 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:44:13.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:12 smithi016 ceph-mon[31772]: pgmap v293: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-13T03:44:13.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:12 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:44:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:12 smithi188 ceph-mon[36398]: pgmap v293: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 187 op/s 2024-04-13T03:44:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:12 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:44:13.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306983 3.40 MB/sec execute 354 sec latency 1978.458 ms 2024-04-13T03:44:14.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:14 smithi016 ceph-mon[31772]: osdmap e103: 12 total, 12 up, 12 in 2024-04-13T03:44:14.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:14 smithi016 ceph-mon[31772]: pgmap v295: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 234 op/s 2024-04-13T03:44:14.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:14 smithi188 ceph-mon[36398]: osdmap e103: 12 total, 12 up, 12 in 2024-04-13T03:44:14.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:14 smithi188 ceph-mon[36398]: pgmap v295: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 234 op/s 2024-04-13T03:44:14.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:14 smithi119 ceph-mon[35835]: osdmap e103: 12 total, 12 up, 12 in 2024-04-13T03:44:14.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:14 smithi119 ceph-mon[35835]: pgmap v295: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 234 op/s 2024-04-13T03:44:14.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306983 3.39 MB/sec execute 355 sec latency 2978.604 ms 2024-04-13T03:44:15.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 306983 3.38 MB/sec execute 356 sec latency 3978.719 ms 2024-04-13T03:44:16.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 307390 3.38 MB/sec execute 357 sec latency 4804.326 ms 2024-04-13T03:44:17.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:16 smithi119 ceph-mon[35835]: pgmap v296: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 847 KiB/s wr, 76 op/s 2024-04-13T03:44:17.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:16 smithi016 ceph-mon[31772]: pgmap v296: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 847 KiB/s wr, 76 op/s 2024-04-13T03:44:17.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:16 smithi188 ceph-mon[36398]: pgmap v296: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 847 KiB/s wr, 76 op/s 2024-04-13T03:44:17.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 308607 3.40 MB/sec execute 358 sec latency 27.964 ms 2024-04-13T03:44:18.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 309207 3.39 MB/sec execute 359 sec latency 31.273 ms 2024-04-13T03:44:19.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:18 smithi119 ceph-mon[35835]: pgmap v297: 129 pgs: 129 active+clean; 235 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-04-13T03:44:19.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:18 smithi016 ceph-mon[31772]: pgmap v297: 129 pgs: 129 active+clean; 235 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-04-13T03:44:19.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:18 smithi188 ceph-mon[36398]: pgmap v297: 129 pgs: 129 active+clean; 235 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-04-13T03:44:19.930 INFO:tasks.workunit.client.0.smithi016.stdout: 1 310082 3.40 MB/sec execute 360 sec latency 21.276 ms 2024-04-13T03:44:20.931 INFO:tasks.workunit.client.0.smithi016.stdout: 1 310305 3.39 MB/sec execute 361 sec latency 580.128 ms 2024-04-13T03:44:21.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:20 smithi119 ceph-mon[35835]: pgmap v298: 129 pgs: 129 active+clean; 235 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-04-13T03:44:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:20 smithi016 ceph-mon[31772]: pgmap v298: 129 pgs: 129 active+clean; 235 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-04-13T03:44:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:20 smithi188 ceph-mon[36398]: pgmap v298: 129 pgs: 129 active+clean; 235 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-04-13T03:44:21.931 INFO:tasks.workunit.client.0.smithi016.stdout: 1 310505 3.38 MB/sec execute 362 sec latency 1414.558 ms 2024-04-13T03:44:22.931 INFO:tasks.workunit.client.0.smithi016.stdout: 1 311821 3.39 MB/sec execute 363 sec latency 26.750 ms 2024-04-13T03:44:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:22 smithi119 ceph-mon[35835]: pgmap v299: 129 pgs: 129 active+clean; 236 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 892 KiB/s wr, 91 op/s 2024-04-13T03:44:23.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:22 smithi016 ceph-mon[31772]: pgmap v299: 129 pgs: 129 active+clean; 236 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 892 KiB/s wr, 91 op/s 2024-04-13T03:44:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:22 smithi188 ceph-mon[36398]: pgmap v299: 129 pgs: 129 active+clean; 236 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 892 KiB/s wr, 91 op/s 2024-04-13T03:44:23.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 312616 3.40 MB/sec execute 364 sec latency 17.519 ms 2024-04-13T03:44:24.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:24 smithi016 ceph-mon[31772]: pgmap v300: 129 pgs: 129 active+clean; 236 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 842 KiB/s wr, 86 op/s 2024-04-13T03:44:24.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:24 smithi188 ceph-mon[36398]: pgmap v300: 129 pgs: 129 active+clean; 236 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 842 KiB/s wr, 86 op/s 2024-04-13T03:44:24.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:24 smithi119 ceph-mon[35835]: pgmap v300: 129 pgs: 129 active+clean; 236 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 842 KiB/s wr, 86 op/s 2024-04-13T03:44:24.931 INFO:tasks.workunit.client.0.smithi016.stdout: 1 313406 3.40 MB/sec execute 365 sec latency 19.244 ms 2024-04-13T03:44:25.932 INFO:tasks.workunit.client.0.smithi016.stdout: 1 313847 3.40 MB/sec execute 366 sec latency 328.302 ms 2024-04-13T03:44:26.932 INFO:tasks.workunit.client.0.smithi016.stdout: 1 314070 3.39 MB/sec execute 367 sec latency 1156.439 ms 2024-04-13T03:44:27.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:26 smithi119 ceph-mon[35835]: pgmap v301: 129 pgs: 129 active+clean; 236 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 744 KiB/s wr, 76 op/s 2024-04-13T03:44:27.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:26 smithi016 ceph-mon[31772]: pgmap v301: 129 pgs: 129 active+clean; 236 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 744 KiB/s wr, 76 op/s 2024-04-13T03:44:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:26 smithi188 ceph-mon[36398]: pgmap v301: 129 pgs: 129 active+clean; 236 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 744 KiB/s wr, 76 op/s 2024-04-13T03:44:27.932 INFO:tasks.workunit.client.0.smithi016.stdout: 1 315342 3.40 MB/sec execute 368 sec latency 24.234 ms 2024-04-13T03:44:28.932 INFO:tasks.workunit.client.0.smithi016.stdout: 1 316128 3.41 MB/sec execute 369 sec latency 16.433 ms 2024-04-13T03:44:29.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:28 smithi119 ceph-mon[35835]: pgmap v302: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 133 op/s 2024-04-13T03:44:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:28 smithi016 ceph-mon[31772]: pgmap v302: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 133 op/s 2024-04-13T03:44:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:28 smithi188 ceph-mon[36398]: pgmap v302: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 133 op/s 2024-04-13T03:44:29.932 INFO:tasks.workunit.client.0.smithi016.stdout: 1 316884 3.40 MB/sec execute 370 sec latency 19.788 ms 2024-04-13T03:44:30.932 INFO:tasks.workunit.client.0.smithi016.stdout: 1 317396 3.40 MB/sec execute 371 sec latency 475.398 ms 2024-04-13T03:44:31.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:30 smithi119 ceph-mon[35835]: pgmap v303: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:44:31.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:30 smithi016 ceph-mon[31772]: pgmap v303: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:44:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:30 smithi188 ceph-mon[36398]: pgmap v303: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-13T03:44:31.932 INFO:tasks.workunit.client.0.smithi016.stdout: 1 317599 3.39 MB/sec execute 372 sec latency 1312.528 ms 2024-04-13T03:44:32.932 INFO:tasks.workunit.client.0.smithi016.stdout: 1 318890 3.40 MB/sec execute 373 sec latency 30.081 ms 2024-04-13T03:44:33.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:32 smithi119 ceph-mon[35835]: pgmap v304: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-13T03:44:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:32 smithi016 ceph-mon[31772]: pgmap v304: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-13T03:44:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:32 smithi188 ceph-mon[36398]: pgmap v304: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-13T03:44:33.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 319642 3.41 MB/sec execute 374 sec latency 24.311 ms 2024-04-13T03:44:34.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 320371 3.41 MB/sec execute 375 sec latency 23.698 ms 2024-04-13T03:44:35.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:34 smithi119 ceph-mon[35835]: pgmap v305: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-13T03:44:35.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:34 smithi016 ceph-mon[31772]: pgmap v305: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-13T03:44:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:34 smithi188 ceph-mon[36398]: pgmap v305: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-13T03:44:35.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 320938 3.41 MB/sec execute 376 sec latency 68.227 ms 2024-04-13T03:44:36.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 321072 3.40 MB/sec execute 377 sec latency 896.174 ms 2024-04-13T03:44:37.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:36 smithi119 ceph-mon[35835]: pgmap v306: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-13T03:44:37.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:36 smithi016 ceph-mon[31772]: pgmap v306: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-13T03:44:37.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:36 smithi188 ceph-mon[36398]: pgmap v306: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-13T03:44:37.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 322419 3.41 MB/sec execute 378 sec latency 24.234 ms 2024-04-13T03:44:38.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 323202 3.42 MB/sec execute 379 sec latency 19.950 ms 2024-04-13T03:44:39.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:38 smithi119 ceph-mon[35835]: pgmap v307: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 175 op/s 2024-04-13T03:44:39.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:38 smithi016 ceph-mon[31772]: pgmap v307: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 175 op/s 2024-04-13T03:44:39.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:38 smithi188 ceph-mon[36398]: pgmap v307: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 175 op/s 2024-04-13T03:44:39.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 323974 3.41 MB/sec execute 380 sec latency 21.425 ms 2024-04-13T03:44:40.933 INFO:tasks.workunit.client.0.smithi016.stdout: 1 324487 3.41 MB/sec execute 381 sec latency 497.473 ms 2024-04-13T03:44:41.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:40 smithi119 ceph-mon[35835]: pgmap v308: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-04-13T03:44:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:40 smithi016 ceph-mon[31772]: pgmap v308: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-04-13T03:44:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:40 smithi188 ceph-mon[36398]: pgmap v308: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 117 op/s 2024-04-13T03:44:41.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 324710 3.40 MB/sec execute 382 sec latency 1324.038 ms 2024-04-13T03:44:42.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 326001 3.42 MB/sec execute 383 sec latency 47.812 ms 2024-04-13T03:44:43.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:42 smithi119 ceph-mon[35835]: pgmap v309: 129 pgs: 129 active+clean; 252 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 178 op/s 2024-04-13T03:44:43.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:42 smithi016 ceph-mon[31772]: pgmap v309: 129 pgs: 129 active+clean; 252 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 178 op/s 2024-04-13T03:44:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:42 smithi188 ceph-mon[36398]: pgmap v309: 129 pgs: 129 active+clean; 252 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 178 op/s 2024-04-13T03:44:43.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 326802 3.42 MB/sec execute 384 sec latency 14.785 ms 2024-04-13T03:44:44.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 327747 3.43 MB/sec execute 385 sec latency 24.369 ms 2024-04-13T03:44:45.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:44 smithi119 ceph-mon[35835]: pgmap v310: 129 pgs: 129 active+clean; 252 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-04-13T03:44:45.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:44 smithi016 ceph-mon[31772]: pgmap v310: 129 pgs: 129 active+clean; 252 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-04-13T03:44:45.160 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:44 smithi188 ceph-mon[36398]: pgmap v310: 129 pgs: 129 active+clean; 252 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-04-13T03:44:45.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 328029 3.42 MB/sec execute 386 sec latency 661.924 ms 2024-04-13T03:44:46.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 328253 3.41 MB/sec execute 387 sec latency 1487.029 ms 2024-04-13T03:44:47.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:46 smithi119 ceph-mon[35835]: pgmap v311: 129 pgs: 129 active+clean; 252 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-04-13T03:44:47.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:46 smithi016 ceph-mon[31772]: pgmap v311: 129 pgs: 129 active+clean; 252 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-04-13T03:44:47.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:46 smithi188 ceph-mon[36398]: pgmap v311: 129 pgs: 129 active+clean; 252 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-04-13T03:44:47.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 329640 3.42 MB/sec execute 388 sec latency 20.900 ms 2024-04-13T03:44:48.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 330467 3.43 MB/sec execute 389 sec latency 12.063 ms 2024-04-13T03:44:49.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:48 smithi119 ceph-mon[35835]: pgmap v312: 129 pgs: 129 active+clean; 256 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 182 op/s 2024-04-13T03:44:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:48 smithi016 ceph-mon[31772]: pgmap v312: 129 pgs: 129 active+clean; 256 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 182 op/s 2024-04-13T03:44:49.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:48 smithi188 ceph-mon[36398]: pgmap v312: 129 pgs: 129 active+clean; 256 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 182 op/s 2024-04-13T03:44:49.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 331445 3.43 MB/sec execute 390 sec latency 22.068 ms 2024-04-13T03:44:50.934 INFO:tasks.workunit.client.0.smithi016.stdout: 1 331577 3.42 MB/sec execute 391 sec latency 847.999 ms 2024-04-13T03:44:51.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:50 smithi119 ceph-mon[35835]: pgmap v313: 129 pgs: 129 active+clean; 256 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 122 op/s 2024-04-13T03:44:51.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:50 smithi016 ceph-mon[31772]: pgmap v313: 129 pgs: 129 active+clean; 256 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 122 op/s 2024-04-13T03:44:51.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:50 smithi188 ceph-mon[36398]: pgmap v313: 129 pgs: 129 active+clean; 256 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 122 op/s 2024-04-13T03:44:51.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 331804 3.41 MB/sec execute 392 sec latency 1674.064 ms 2024-04-13T03:44:52.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 333199 3.43 MB/sec execute 393 sec latency 60.935 ms 2024-04-13T03:44:53.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:52 smithi016 ceph-mon[31772]: pgmap v314: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-04-13T03:44:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:52 smithi188 ceph-mon[36398]: pgmap v314: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-04-13T03:44:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:52 smithi119 ceph-mon[35835]: pgmap v314: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-04-13T03:44:53.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 333989 3.43 MB/sec execute 394 sec latency 11.956 ms 2024-04-13T03:44:54.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 334881 3.44 MB/sec execute 395 sec latency 21.296 ms 2024-04-13T03:44:55.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:54 smithi016 ceph-mon[31772]: pgmap v315: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:44:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:54 smithi188 ceph-mon[36398]: pgmap v315: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:44:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:54 smithi119 ceph-mon[35835]: pgmap v315: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:44:55.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335120 3.43 MB/sec execute 396 sec latency 701.589 ms 2024-04-13T03:44:56.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 335322 3.42 MB/sec execute 397 sec latency 1530.473 ms 2024-04-13T03:44:57.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:56 smithi016 ceph-mon[31772]: pgmap v316: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:44:57.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:56 smithi188 ceph-mon[36398]: pgmap v316: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:44:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:56 smithi119 ceph-mon[35835]: pgmap v316: 129 pgs: 129 active+clean; 260 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:44:57.935 INFO:tasks.workunit.client.0.smithi016.stdout: 1 336596 3.43 MB/sec execute 398 sec latency 47.439 ms 2024-04-13T03:44:58.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 337398 3.44 MB/sec execute 399 sec latency 25.074 ms 2024-04-13T03:44:59.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:44:58 smithi016 ceph-mon[31772]: pgmap v317: 129 pgs: 129 active+clean; 264 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-13T03:44:59.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:44:58 smithi188 ceph-mon[36398]: pgmap v317: 129 pgs: 129 active+clean; 264 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-13T03:44:59.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:44:58 smithi119 ceph-mon[35835]: pgmap v317: 129 pgs: 129 active+clean; 264 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-13T03:44:59.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338209 3.43 MB/sec execute 400 sec latency 18.553 ms 2024-04-13T03:45:00.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338670 3.43 MB/sec execute 401 sec latency 559.185 ms 2024-04-13T03:45:01.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:00 smithi016 ceph-mon[31772]: pgmap v318: 129 pgs: 129 active+clean; 264 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:45:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:00 smithi188 ceph-mon[36398]: pgmap v318: 129 pgs: 129 active+clean; 264 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:45:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:00 smithi119 ceph-mon[35835]: pgmap v318: 129 pgs: 129 active+clean; 264 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 113 op/s 2024-04-13T03:45:01.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 338839 3.42 MB/sec execute 402 sec latency 1386.324 ms 2024-04-13T03:45:02.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339265 3.42 MB/sec execute 403 sec latency 152.473 ms 2024-04-13T03:45:03.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:02 smithi016 ceph-mon[31772]: pgmap v319: 129 pgs: 129 active+clean; 267 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-13T03:45:03.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:02 smithi188 ceph-mon[36398]: pgmap v319: 129 pgs: 129 active+clean; 267 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-13T03:45:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:02 smithi119 ceph-mon[35835]: pgmap v319: 129 pgs: 129 active+clean; 267 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 174 op/s 2024-04-13T03:45:03.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 339882 3.42 MB/sec execute 404 sec latency 83.660 ms 2024-04-13T03:45:04.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 340881 3.43 MB/sec execute 405 sec latency 25.546 ms 2024-04-13T03:45:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:04 smithi188 ceph-mon[36398]: pgmap v320: 129 pgs: 129 active+clean; 267 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:45:05.123 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:04 smithi119 ceph-mon[35835]: pgmap v320: 129 pgs: 129 active+clean; 267 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:45:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:04 smithi016 ceph-mon[31772]: pgmap v320: 129 pgs: 129 active+clean; 267 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:45:05.936 INFO:tasks.workunit.client.0.smithi016.stdout: 1 341591 3.43 MB/sec execute 406 sec latency 20.350 ms 2024-04-13T03:45:06.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 342427 3.43 MB/sec execute 407 sec latency 243.928 ms 2024-04-13T03:45:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:06 smithi188 ceph-mon[36398]: pgmap v321: 129 pgs: 129 active+clean; 267 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:45:07.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:06 smithi119 ceph-mon[35835]: pgmap v321: 129 pgs: 129 active+clean; 267 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:45:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:06 smithi016 ceph-mon[31772]: pgmap v321: 129 pgs: 129 active+clean; 267 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:45:07.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 343856 3.44 MB/sec execute 408 sec latency 32.031 ms 2024-04-13T03:45:08.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:07 smithi119 ceph-mon[35835]: osdmap e104: 12 total, 12 up, 12 in 2024-04-13T03:45:08.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:07 smithi016 ceph-mon[31772]: osdmap e104: 12 total, 12 up, 12 in 2024-04-13T03:45:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:07 smithi188 ceph-mon[36398]: osdmap e104: 12 total, 12 up, 12 in 2024-04-13T03:45:08.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 344534 3.45 MB/sec execute 409 sec latency 19.270 ms 2024-04-13T03:45:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:08 smithi119 ceph-mon[35835]: pgmap v323: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-13T03:45:09.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:08 smithi016 ceph-mon[31772]: pgmap v323: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-13T03:45:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:08 smithi188 ceph-mon[36398]: pgmap v323: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-13T03:45:09.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345452 3.45 MB/sec execute 410 sec latency 22.154 ms 2024-04-13T03:45:10.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:09 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:45:10.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:09 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:45:10.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:09 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:45:10.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345759 3.44 MB/sec execute 411 sec latency 630.920 ms 2024-04-13T03:45:11.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:10 smithi188 ceph-mon[36398]: pgmap v324: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-13T03:45:11.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:10 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:10 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:10 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:10 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:10 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.223 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:10 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:10 smithi119 ceph-mon[35835]: pgmap v324: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-13T03:45:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:10 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:10 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:10 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:10 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:10 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:10 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:10 smithi016 ceph-mon[31772]: pgmap v324: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 187 op/s 2024-04-13T03:45:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:10 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:10 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:10 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:10 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:10 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:10 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:11.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.43 MB/sec execute 412 sec latency 1453.234 ms 2024-04-13T03:45:12.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.43 MB/sec execute 413 sec latency 1177.755 ms 2024-04-13T03:45:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: pgmap v325: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 188 op/s 2024-04-13T03:45:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:12 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: pgmap v325: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 188 op/s 2024-04-13T03:45:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:12 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: pgmap v325: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 188 op/s 2024-04-13T03:45:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:12 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:13.937 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.42 MB/sec execute 414 sec latency 2177.853 ms 2024-04-13T03:45:14.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi188 to 174.1M 2024-04-13T03:45:14.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: Unable to set osd_memory_target on smithi188 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:45:14.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: osdmap e105: 12 total, 12 up, 12 in 2024-04-13T03:45:14.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:14.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:13 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi188 to 174.1M 2024-04-13T03:45:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: Unable to set osd_memory_target on smithi188 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:45:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: osdmap e105: 12 total, 12 up, 12 in 2024-04-13T03:45:14.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:14.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:13 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:14.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi188 to 174.1M 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: Unable to set osd_memory_target on smithi188 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: osdmap e105: 12 total, 12 up, 12 in 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:14.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:13 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:14.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.41 MB/sec execute 415 sec latency 3177.946 ms 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi119 to 174.1M 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: Unable to set osd_memory_target on smithi119 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: pgmap v327: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 235 op/s 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:45:15.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:14 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:45:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi119 to 174.1M 2024-04-13T03:45:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: Unable to set osd_memory_target on smithi119 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:45:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: pgmap v327: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 235 op/s 2024-04-13T03:45:15.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:45:15.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:14 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:45:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi119 to 174.1M 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: Unable to set osd_memory_target on smithi119 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: pgmap v327: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 235 op/s 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:45:15.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:14 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:45:15.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.40 MB/sec execute 416 sec latency 4178.084 ms 2024-04-13T03:45:16.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:15 smithi119 ceph-mon[35835]: Adjusting osd_memory_target on smithi016 to 177.5M 2024-04-13T03:45:16.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:15 smithi119 ceph-mon[35835]: Unable to set osd_memory_target on smithi016 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-13T03:45:16.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:15 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:16.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:15 smithi016 ceph-mon[31772]: Adjusting osd_memory_target on smithi016 to 177.5M 2024-04-13T03:45:16.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:15 smithi016 ceph-mon[31772]: Unable to set osd_memory_target on smithi016 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-13T03:45:16.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:15 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:16.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:15 smithi188 ceph-mon[36398]: Adjusting osd_memory_target on smithi016 to 177.5M 2024-04-13T03:45:16.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:15 smithi188 ceph-mon[36398]: Unable to set osd_memory_target on smithi016 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-13T03:45:16.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:15 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:45:16.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.39 MB/sec execute 417 sec latency 5178.229 ms 2024-04-13T03:45:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:16 smithi119 ceph-mon[35835]: pgmap v328: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 937 KiB/s wr, 82 op/s 2024-04-13T03:45:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:16 smithi016 ceph-mon[31772]: pgmap v328: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 937 KiB/s wr, 82 op/s 2024-04-13T03:45:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:16 smithi188 ceph-mon[36398]: pgmap v328: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 937 KiB/s wr, 82 op/s 2024-04-13T03:45:17.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.38 MB/sec execute 418 sec latency 6178.396 ms 2024-04-13T03:45:18.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.38 MB/sec execute 419 sec latency 7178.537 ms 2024-04-13T03:45:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:18 smithi119 ceph-mon[35835]: pgmap v329: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-13T03:45:19.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:18 smithi016 ceph-mon[31772]: pgmap v329: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-13T03:45:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:18 smithi188 ceph-mon[36398]: pgmap v329: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-13T03:45:19.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.37 MB/sec execute 420 sec latency 8178.637 ms 2024-04-13T03:45:20.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.36 MB/sec execute 421 sec latency 9178.777 ms 2024-04-13T03:45:21.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:20 smithi119 ceph-mon[35835]: pgmap v330: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-13T03:45:21.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:20 smithi016 ceph-mon[31772]: pgmap v330: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-13T03:45:21.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:20 smithi188 ceph-mon[36398]: pgmap v330: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-13T03:45:21.938 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.35 MB/sec execute 422 sec latency 10178.888 ms 2024-04-13T03:45:22.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:21 smithi119 ceph-mon[35835]: pgmap v331: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 840 KiB/s wr, 49 op/s 2024-04-13T03:45:22.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:21 smithi016 ceph-mon[31772]: pgmap v331: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 840 KiB/s wr, 49 op/s 2024-04-13T03:45:22.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:21 smithi188 ceph-mon[36398]: pgmap v331: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 840 KiB/s wr, 49 op/s 2024-04-13T03:45:22.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345762 3.34 MB/sec execute 423 sec latency 11179.019 ms 2024-04-13T03:45:23.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 345771 3.34 MB/sec execute 424 sec latency 12168.121 ms 2024-04-13T03:45:24.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 346996 3.35 MB/sec execute 425 sec latency 28.413 ms 2024-04-13T03:45:25.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:24 smithi119 ceph-mon[35835]: pgmap v332: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 793 KiB/s wr, 47 op/s 2024-04-13T03:45:25.075 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:24 smithi188 ceph-mon[36398]: pgmap v332: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 793 KiB/s wr, 47 op/s 2024-04-13T03:45:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:24 smithi016 ceph-mon[31772]: pgmap v332: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 793 KiB/s wr, 47 op/s 2024-04-13T03:45:25.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 347931 3.35 MB/sec execute 426 sec latency 19.890 ms 2024-04-13T03:45:26.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 348590 3.35 MB/sec execute 427 sec latency 26.423 ms 2024-04-13T03:45:27.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:26 smithi119 ceph-mon[35835]: pgmap v333: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 700 KiB/s wr, 41 op/s 2024-04-13T03:45:27.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:26 smithi016 ceph-mon[31772]: pgmap v333: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 700 KiB/s wr, 41 op/s 2024-04-13T03:45:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:26 smithi188 ceph-mon[36398]: pgmap v333: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 700 KiB/s wr, 41 op/s 2024-04-13T03:45:27.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349302 3.35 MB/sec execute 428 sec latency 56.389 ms 2024-04-13T03:45:28.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:27 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:28.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:27 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:28.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:27 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:28.939 INFO:tasks.workunit.client.0.smithi016.stdout: 1 349305 3.34 MB/sec execute 429 sec latency 1037.676 ms 2024-04-13T03:45:29.073 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:28 smithi119 ceph-mon[35835]: pgmap v334: 129 pgs: 129 active+clean; 278 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.2 MiB/s wr, 86 op/s 2024-04-13T03:45:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:28 smithi016 ceph-mon[31772]: pgmap v334: 129 pgs: 129 active+clean; 278 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.2 MiB/s wr, 86 op/s 2024-04-13T03:45:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:28 smithi188 ceph-mon[36398]: pgmap v334: 129 pgs: 129 active+clean; 278 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 1.2 MiB/s wr, 86 op/s 2024-04-13T03:45:29.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 350600 3.35 MB/sec execute 430 sec latency 35.124 ms 2024-04-13T03:45:30.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 351568 3.36 MB/sec execute 431 sec latency 19.776 ms 2024-04-13T03:45:31.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:30 smithi119 ceph-mon[35835]: pgmap v335: 129 pgs: 129 active+clean; 278 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 841 KiB/s wr, 64 op/s 2024-04-13T03:45:31.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:30 smithi016 ceph-mon[31772]: pgmap v335: 129 pgs: 129 active+clean; 278 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 841 KiB/s wr, 64 op/s 2024-04-13T03:45:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:30 smithi188 ceph-mon[36398]: pgmap v335: 129 pgs: 129 active+clean; 278 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 841 KiB/s wr, 64 op/s 2024-04-13T03:45:31.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352336 3.36 MB/sec execute 432 sec latency 23.930 ms 2024-04-13T03:45:32.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352850 3.36 MB/sec execute 433 sec latency 286.627 ms 2024-04-13T03:45:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:32 smithi016 ceph-mon[31772]: pgmap v336: 129 pgs: 129 active+clean; 283 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 1.6 MiB/s wr, 127 op/s 2024-04-13T03:45:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:32 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:32 smithi188 ceph-mon[36398]: pgmap v336: 129 pgs: 129 active+clean; 283 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 1.6 MiB/s wr, 127 op/s 2024-04-13T03:45:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:32 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:33.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:32 smithi119 ceph-mon[35835]: pgmap v336: 129 pgs: 129 active+clean; 283 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 1.6 MiB/s wr, 127 op/s 2024-04-13T03:45:33.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:32 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:33.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 352853 3.35 MB/sec execute 434 sec latency 1266.781 ms 2024-04-13T03:45:34.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 354180 3.36 MB/sec execute 435 sec latency 24.506 ms 2024-04-13T03:45:35.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:34 smithi016 ceph-mon[31772]: pgmap v337: 129 pgs: 129 active+clean; 283 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:45:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:34 smithi188 ceph-mon[36398]: pgmap v337: 129 pgs: 129 active+clean; 283 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:45:35.122 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:34 smithi119 ceph-mon[35835]: pgmap v337: 129 pgs: 129 active+clean; 283 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:45:35.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 355178 3.36 MB/sec execute 436 sec latency 15.617 ms 2024-04-13T03:45:36.940 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356135 3.37 MB/sec execute 437 sec latency 24.103 ms 2024-04-13T03:45:37.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:36 smithi016 ceph-mon[31772]: pgmap v338: 129 pgs: 129 active+clean; 283 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:45:37.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:36 smithi188 ceph-mon[36398]: pgmap v338: 129 pgs: 129 active+clean; 283 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:45:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:36 smithi119 ceph-mon[35835]: pgmap v338: 129 pgs: 129 active+clean; 283 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:45:37.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 356392 3.36 MB/sec execute 438 sec latency 554.081 ms 2024-04-13T03:45:38.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:37 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:38.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:37 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:38.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:37 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:38.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 357615 3.37 MB/sec execute 439 sec latency 724.746 ms 2024-04-13T03:45:39.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:38 smithi188 ceph-mon[36398]: pgmap v339: 129 pgs: 129 active+clean; 288 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-04-13T03:45:39.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:38 smithi119 ceph-mon[35835]: pgmap v339: 129 pgs: 129 active+clean; 288 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-04-13T03:45:39.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:38 smithi016 ceph-mon[31772]: pgmap v339: 129 pgs: 129 active+clean; 288 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-04-13T03:45:39.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 358627 3.38 MB/sec execute 440 sec latency 24.353 ms 2024-04-13T03:45:40.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359404 3.37 MB/sec execute 441 sec latency 26.951 ms 2024-04-13T03:45:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:40 smithi188 ceph-mon[36398]: pgmap v340: 129 pgs: 129 active+clean; 288 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:45:41.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:40 smithi119 ceph-mon[35835]: pgmap v340: 129 pgs: 129 active+clean; 288 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:45:41.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:40 smithi016 ceph-mon[31772]: pgmap v340: 129 pgs: 129 active+clean; 288 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-04-13T03:45:41.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359941 3.37 MB/sec execute 442 sec latency 182.190 ms 2024-04-13T03:45:42.941 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359941 3.37 MB/sec execute 443 sec latency 1182.407 ms 2024-04-13T03:45:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:42 smithi188 ceph-mon[36398]: pgmap v341: 129 pgs: 129 active+clean; 293 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 193 op/s 2024-04-13T03:45:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:42 smithi119 ceph-mon[35835]: pgmap v341: 129 pgs: 129 active+clean; 293 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 193 op/s 2024-04-13T03:45:43.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:42 smithi016 ceph-mon[31772]: pgmap v341: 129 pgs: 129 active+clean; 293 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 193 op/s 2024-04-13T03:45:43.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 359944 3.36 MB/sec execute 444 sec latency 2161.499 ms 2024-04-13T03:45:44.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 361015 3.36 MB/sec execute 445 sec latency 23.092 ms 2024-04-13T03:45:45.157 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:44 smithi016 ceph-mon[31772]: pgmap v342: 129 pgs: 129 active+clean; 293 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-04-13T03:45:45.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:44 smithi119 ceph-mon[35835]: pgmap v342: 129 pgs: 129 active+clean; 293 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-04-13T03:45:45.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:44 smithi188 ceph-mon[36398]: pgmap v342: 129 pgs: 129 active+clean; 293 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-04-13T03:45:45.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 362113 3.37 MB/sec execute 446 sec latency 15.204 ms 2024-04-13T03:45:46.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:45 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:46.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:45 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:46.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:45 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:45:46.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 362778 3.37 MB/sec execute 447 sec latency 18.194 ms 2024-04-13T03:45:47.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:46 smithi119 ceph-mon[35835]: pgmap v343: 129 pgs: 129 active+clean; 293 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-04-13T03:45:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:46 smithi016 ceph-mon[31772]: pgmap v343: 129 pgs: 129 active+clean; 293 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-04-13T03:45:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:46 smithi188 ceph-mon[36398]: pgmap v343: 129 pgs: 129 active+clean; 293 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 130 op/s 2024-04-13T03:45:47.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 363483 3.37 MB/sec execute 448 sec latency 424.569 ms 2024-04-13T03:45:48.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 363495 3.36 MB/sec execute 449 sec latency 1400.621 ms 2024-04-13T03:45:49.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:48 smithi119 ceph-mon[35835]: pgmap v344: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 173 op/s 2024-04-13T03:45:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:48 smithi016 ceph-mon[31772]: pgmap v344: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 173 op/s 2024-04-13T03:45:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:48 smithi188 ceph-mon[36398]: pgmap v344: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 173 op/s 2024-04-13T03:45:49.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 365054 3.38 MB/sec execute 450 sec latency 16.469 ms 2024-04-13T03:45:50.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 365874 3.38 MB/sec execute 451 sec latency 14.365 ms 2024-04-13T03:45:51.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:50 smithi119 ceph-mon[35835]: pgmap v345: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-13T03:45:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:50 smithi016 ceph-mon[31772]: pgmap v345: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-13T03:45:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:50 smithi188 ceph-mon[36398]: pgmap v345: 129 pgs: 129 active+clean; 296 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 114 op/s 2024-04-13T03:45:51.942 INFO:tasks.workunit.client.0.smithi016.stdout: 1 366951 3.38 MB/sec execute 452 sec latency 22.637 ms 2024-04-13T03:45:52.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367032 3.38 MB/sec execute 453 sec latency 925.205 ms 2024-04-13T03:45:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:52 smithi119 ceph-mon[35835]: pgmap v346: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 171 op/s 2024-04-13T03:45:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:52 smithi016 ceph-mon[31772]: pgmap v346: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 171 op/s 2024-04-13T03:45:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:52 smithi188 ceph-mon[36398]: pgmap v346: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 171 op/s 2024-04-13T03:45:53.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 367038 3.37 MB/sec execute 454 sec latency 1902.405 ms 2024-04-13T03:45:54.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 368527 3.38 MB/sec execute 455 sec latency 18.021 ms 2024-04-13T03:45:55.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:54 smithi188 ceph-mon[36398]: pgmap v347: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 100 op/s 2024-04-13T03:45:55.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:54 smithi119 ceph-mon[35835]: pgmap v347: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 100 op/s 2024-04-13T03:45:55.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:54 smithi016 ceph-mon[31772]: pgmap v347: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 100 op/s 2024-04-13T03:45:55.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 369332 3.38 MB/sec execute 456 sec latency 16.035 ms 2024-04-13T03:45:56.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370167 3.38 MB/sec execute 457 sec latency 23.430 ms 2024-04-13T03:45:57.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:56 smithi119 ceph-mon[35835]: pgmap v348: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 100 op/s 2024-04-13T03:45:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:56 smithi016 ceph-mon[31772]: pgmap v348: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 100 op/s 2024-04-13T03:45:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:56 smithi188 ceph-mon[36398]: pgmap v348: 129 pgs: 129 active+clean; 301 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 100 op/s 2024-04-13T03:45:57.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 370575 3.38 MB/sec execute 458 sec latency 607.617 ms 2024-04-13T03:45:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: 1 371661 3.39 MB/sec execute 459 sec latency 777.271 ms 2024-04-13T03:45:59.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:45:58 smithi119 ceph-mon[35835]: pgmap v349: 129 pgs: 129 active+clean; 305 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 146 op/s 2024-04-13T03:45:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:45:58 smithi016 ceph-mon[31772]: pgmap v349: 129 pgs: 129 active+clean; 305 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 146 op/s 2024-04-13T03:45:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:45:58 smithi188 ceph-mon[36398]: pgmap v349: 129 pgs: 129 active+clean; 305 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 146 op/s 2024-04-13T03:45:59.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 372756 3.40 MB/sec execute 460 sec latency 19.953 ms 2024-04-13T03:46:00.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 373355 3.39 MB/sec execute 461 sec latency 26.034 ms 2024-04-13T03:46:01.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:00 smithi119 ceph-mon[35835]: pgmap v350: 129 pgs: 129 active+clean; 305 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 916 KiB/s wr, 102 op/s 2024-04-13T03:46:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:00 smithi016 ceph-mon[31772]: pgmap v350: 129 pgs: 129 active+clean; 305 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 916 KiB/s wr, 102 op/s 2024-04-13T03:46:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:00 smithi188 ceph-mon[36398]: pgmap v350: 129 pgs: 129 active+clean; 305 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 916 KiB/s wr, 102 op/s 2024-04-13T03:46:01.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 373463 3.39 MB/sec execute 462 sec latency 96.419 ms 2024-04-13T03:46:02.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:01 smithi119 ceph-mon[35835]: pgmap v351: 129 pgs: 129 active+clean; 310 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 188 op/s 2024-04-13T03:46:02.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:01 smithi016 ceph-mon[31772]: pgmap v351: 129 pgs: 129 active+clean; 310 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 188 op/s 2024-04-13T03:46:02.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:01 smithi188 ceph-mon[36398]: pgmap v351: 129 pgs: 129 active+clean; 310 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 188 op/s 2024-04-13T03:46:02.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 374018 3.39 MB/sec execute 463 sec latency 102.973 ms 2024-04-13T03:46:03.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 374126 3.38 MB/sec execute 464 sec latency 879.642 ms 2024-04-13T03:46:04.944 INFO:tasks.workunit.client.0.smithi016.stdout: 1 375413 3.39 MB/sec execute 465 sec latency 42.134 ms 2024-04-13T03:46:05.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:04 smithi119 ceph-mon[35835]: pgmap v352: 129 pgs: 129 active+clean; 310 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-13T03:46:05.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:04 smithi016 ceph-mon[31772]: pgmap v352: 129 pgs: 129 active+clean; 310 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-13T03:46:05.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:04 smithi188 ceph-mon[36398]: pgmap v352: 129 pgs: 129 active+clean; 310 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-13T03:46:05.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376353 3.39 MB/sec execute 466 sec latency 17.595 ms 2024-04-13T03:46:06.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 376931 3.39 MB/sec execute 467 sec latency 45.827 ms 2024-04-13T03:46:07.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:06 smithi119 ceph-mon[35835]: pgmap v353: 129 pgs: 129 active+clean; 310 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-13T03:46:07.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:06 smithi016 ceph-mon[31772]: pgmap v353: 129 pgs: 129 active+clean; 310 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-13T03:46:07.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:06 smithi188 ceph-mon[36398]: pgmap v353: 129 pgs: 129 active+clean; 310 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 132 op/s 2024-04-13T03:46:07.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 377666 3.39 MB/sec execute 468 sec latency 272.750 ms 2024-04-13T03:46:08.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:07 smithi016 ceph-mon[31772]: osdmap e106: 12 total, 12 up, 12 in 2024-04-13T03:46:08.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:07 smithi188 ceph-mon[36398]: osdmap e106: 12 total, 12 up, 12 in 2024-04-13T03:46:08.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:07 smithi119 ceph-mon[35835]: osdmap e106: 12 total, 12 up, 12 in 2024-04-13T03:46:08.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 377678 3.38 MB/sec execute 469 sec latency 1248.235 ms 2024-04-13T03:46:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:08 smithi016 ceph-mon[31772]: pgmap v355: 129 pgs: 129 active+clean; 315 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 184 op/s 2024-04-13T03:46:09.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:08 smithi188 ceph-mon[36398]: pgmap v355: 129 pgs: 129 active+clean; 315 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 184 op/s 2024-04-13T03:46:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:08 smithi119 ceph-mon[35835]: pgmap v355: 129 pgs: 129 active+clean; 315 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 184 op/s 2024-04-13T03:46:09.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 379024 3.39 MB/sec execute 470 sec latency 60.807 ms 2024-04-13T03:46:10.945 INFO:tasks.workunit.client.0.smithi016.stdout: 1 379968 3.40 MB/sec execute 471 sec latency 21.733 ms 2024-04-13T03:46:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:10 smithi016 ceph-mon[31772]: pgmap v356: 129 pgs: 129 active+clean; 315 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 184 op/s 2024-04-13T03:46:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:10 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:46:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:10 smithi188 ceph-mon[36398]: pgmap v356: 129 pgs: 129 active+clean; 315 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 184 op/s 2024-04-13T03:46:11.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:10 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:46:11.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:10 smithi119 ceph-mon[35835]: pgmap v356: 129 pgs: 129 active+clean; 315 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 184 op/s 2024-04-13T03:46:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:10 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:46:11.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 380622 3.40 MB/sec execute 472 sec latency 24.133 ms 2024-04-13T03:46:12.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381255 3.40 MB/sec execute 473 sec latency 328.334 ms 2024-04-13T03:46:13.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:12 smithi016 ceph-mon[31772]: pgmap v357: 129 pgs: 129 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 157 op/s 2024-04-13T03:46:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:12 smithi188 ceph-mon[36398]: pgmap v357: 129 pgs: 129 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 157 op/s 2024-04-13T03:46:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:12 smithi119 ceph-mon[35835]: pgmap v357: 129 pgs: 129 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 157 op/s 2024-04-13T03:46:13.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 381258 3.39 MB/sec execute 474 sec latency 1308.667 ms 2024-04-13T03:46:14.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:14 smithi119 ceph-mon[35835]: osdmap e107: 12 total, 12 up, 12 in 2024-04-13T03:46:14.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:14 smithi119 ceph-mon[35835]: pgmap v359: 129 pgs: 129 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 197 op/s 2024-04-13T03:46:14.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:14 smithi016 ceph-mon[31772]: osdmap e107: 12 total, 12 up, 12 in 2024-04-13T03:46:14.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:14 smithi016 ceph-mon[31772]: pgmap v359: 129 pgs: 129 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 197 op/s 2024-04-13T03:46:14.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:14 smithi188 ceph-mon[36398]: osdmap e107: 12 total, 12 up, 12 in 2024-04-13T03:46:14.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:14 smithi188 ceph-mon[36398]: pgmap v359: 129 pgs: 129 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 197 op/s 2024-04-13T03:46:14.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 382830 3.40 MB/sec execute 475 sec latency 27.036 ms 2024-04-13T03:46:15.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:15 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:46:15.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:15 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:46:15.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:15 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:46:15.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 383665 3.40 MB/sec execute 476 sec latency 13.524 ms 2024-04-13T03:46:16.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:16 smithi119 ceph-mon[35835]: pgmap v360: 129 pgs: 129 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 1.2 MiB/s wr, 86 op/s 2024-04-13T03:46:16.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:16 smithi016 ceph-mon[31772]: pgmap v360: 129 pgs: 129 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 1.2 MiB/s wr, 86 op/s 2024-04-13T03:46:16.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:16 smithi188 ceph-mon[36398]: pgmap v360: 129 pgs: 129 active+clean; 317 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 1.2 MiB/s wr, 86 op/s 2024-04-13T03:46:16.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384599 3.41 MB/sec execute 477 sec latency 26.078 ms 2024-04-13T03:46:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:17 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:46:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:17 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:46:17.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:17 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:46:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:17 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:46:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:17 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:46:17.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:17 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:46:17.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:17 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:46:17.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:17 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:46:17.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:17 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:46:17.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 384796 3.40 MB/sec execute 478 sec latency 680.414 ms 2024-04-13T03:46:18.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:18 smithi016 ceph-mon[31772]: pgmap v361: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-04-13T03:46:18.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:18 smithi188 ceph-mon[36398]: pgmap v361: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-04-13T03:46:18.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:18 smithi119 ceph-mon[35835]: pgmap v361: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-04-13T03:46:18.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 385895 3.40 MB/sec execute 479 sec latency 850.050 ms 2024-04-13T03:46:19.946 INFO:tasks.workunit.client.0.smithi016.stdout: 1 387052 3.41 MB/sec execute 480 sec latency 16.177 ms 2024-04-13T03:46:20.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 387802 3.41 MB/sec execute 481 sec latency 26.999 ms 2024-04-13T03:46:21.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:20 smithi119 ceph-mon[35835]: pgmap v362: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-04-13T03:46:21.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:20 smithi016 ceph-mon[31772]: pgmap v362: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-04-13T03:46:21.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:20 smithi188 ceph-mon[36398]: pgmap v362: 129 pgs: 129 active+clean; 323 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-04-13T03:46:21.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388368 3.41 MB/sec execute 482 sec latency 431.120 ms 2024-04-13T03:46:22.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388368 3.40 MB/sec execute 483 sec latency 1431.286 ms 2024-04-13T03:46:23.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:22 smithi119 ceph-mon[35835]: pgmap v363: 129 pgs: 129 active+clean; 328 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-04-13T03:46:23.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:22 smithi016 ceph-mon[31772]: pgmap v363: 129 pgs: 129 active+clean; 328 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-04-13T03:46:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:22 smithi188 ceph-mon[36398]: pgmap v363: 129 pgs: 129 active+clean; 328 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-04-13T03:46:23.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 388380 3.40 MB/sec execute 484 sec latency 2405.862 ms 2024-04-13T03:46:24.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:24 smithi119 ceph-mon[35835]: pgmap v364: 129 pgs: 129 active+clean; 328 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-04-13T03:46:24.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:24 smithi016 ceph-mon[31772]: pgmap v364: 129 pgs: 129 active+clean; 328 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-04-13T03:46:24.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:24 smithi188 ceph-mon[36398]: pgmap v364: 129 pgs: 129 active+clean; 328 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-04-13T03:46:24.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 389844 3.41 MB/sec execute 485 sec latency 54.905 ms 2024-04-13T03:46:25.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 390676 3.41 MB/sec execute 486 sec latency 16.167 ms 2024-04-13T03:46:26.947 INFO:tasks.workunit.client.0.smithi016.stdout: 1 391452 3.41 MB/sec execute 487 sec latency 30.602 ms 2024-04-13T03:46:27.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:26 smithi119 ceph-mon[35835]: pgmap v365: 129 pgs: 129 active+clean; 328 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-04-13T03:46:27.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:26 smithi016 ceph-mon[31772]: pgmap v365: 129 pgs: 129 active+clean; 328 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-04-13T03:46:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:26 smithi188 ceph-mon[36398]: pgmap v365: 129 pgs: 129 active+clean; 328 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-04-13T03:46:27.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 391911 3.41 MB/sec execute 488 sec latency 555.816 ms 2024-04-13T03:46:28.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 391914 3.40 MB/sec execute 489 sec latency 1533.776 ms 2024-04-13T03:46:29.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:28 smithi119 ceph-mon[35835]: pgmap v366: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 182 op/s 2024-04-13T03:46:29.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:28 smithi016 ceph-mon[31772]: pgmap v366: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 182 op/s 2024-04-13T03:46:29.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:28 smithi188 ceph-mon[36398]: pgmap v366: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 182 op/s 2024-04-13T03:46:29.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 393228 3.41 MB/sec execute 490 sec latency 37.799 ms 2024-04-13T03:46:30.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 394193 3.42 MB/sec execute 491 sec latency 35.615 ms 2024-04-13T03:46:31.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:30 smithi119 ceph-mon[35835]: pgmap v367: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-13T03:46:31.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:30 smithi016 ceph-mon[31772]: pgmap v367: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-13T03:46:31.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:30 smithi188 ceph-mon[36398]: pgmap v367: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 116 op/s 2024-04-13T03:46:31.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395147 3.42 MB/sec execute 492 sec latency 20.148 ms 2024-04-13T03:46:32.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395459 3.41 MB/sec execute 493 sec latency 635.189 ms 2024-04-13T03:46:33.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:32 smithi016 ceph-mon[31772]: pgmap v368: 129 pgs: 129 active+clean; 336 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 180 op/s 2024-04-13T03:46:33.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:32 smithi188 ceph-mon[36398]: pgmap v368: 129 pgs: 129 active+clean; 336 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 180 op/s 2024-04-13T03:46:33.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:32 smithi119 ceph-mon[35835]: pgmap v368: 129 pgs: 129 active+clean; 336 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 180 op/s 2024-04-13T03:46:33.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 395462 3.41 MB/sec execute 494 sec latency 1621.356 ms 2024-04-13T03:46:34.948 INFO:tasks.workunit.client.0.smithi016.stdout: 1 397033 3.42 MB/sec execute 495 sec latency 20.097 ms 2024-04-13T03:46:35.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:34 smithi016 ceph-mon[31772]: pgmap v369: 129 pgs: 129 active+clean; 336 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-13T03:46:35.122 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:34 smithi119 ceph-mon[35835]: pgmap v369: 129 pgs: 129 active+clean; 336 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-13T03:46:35.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:34 smithi188 ceph-mon[36398]: pgmap v369: 129 pgs: 129 active+clean; 336 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-13T03:46:35.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 397867 3.42 MB/sec execute 496 sec latency 14.253 ms 2024-04-13T03:46:36.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 398745 3.42 MB/sec execute 497 sec latency 26.359 ms 2024-04-13T03:46:37.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:36 smithi016 ceph-mon[31772]: pgmap v370: 129 pgs: 129 active+clean; 336 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-13T03:46:37.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:36 smithi188 ceph-mon[36398]: pgmap v370: 129 pgs: 129 active+clean; 336 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-13T03:46:37.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:36 smithi119 ceph-mon[35835]: pgmap v370: 129 pgs: 129 active+clean; 336 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-13T03:46:37.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 399001 3.42 MB/sec execute 498 sec latency 567.800 ms 2024-04-13T03:46:38.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 400239 3.42 MB/sec execute 499 sec latency 734.880 ms 2024-04-13T03:46:39.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:38 smithi016 ceph-mon[31772]: pgmap v371: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 170 op/s 2024-04-13T03:46:39.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:38 smithi188 ceph-mon[36398]: pgmap v371: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 170 op/s 2024-04-13T03:46:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:38 smithi119 ceph-mon[35835]: pgmap v371: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 170 op/s 2024-04-13T03:46:39.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 401300 3.43 MB/sec execute 500 sec latency 29.053 ms 2024-04-13T03:46:40.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 402283 3.43 MB/sec execute 501 sec latency 24.912 ms 2024-04-13T03:46:41.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:40 smithi016 ceph-mon[31772]: pgmap v372: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 122 op/s 2024-04-13T03:46:41.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:40 smithi188 ceph-mon[36398]: pgmap v372: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 122 op/s 2024-04-13T03:46:41.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:40 smithi119 ceph-mon[35835]: pgmap v372: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 122 op/s 2024-04-13T03:46:41.949 INFO:tasks.workunit.client.0.smithi016.stdout: 1 402551 3.43 MB/sec execute 502 sec latency 699.540 ms 2024-04-13T03:46:42.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 402551 3.42 MB/sec execute 503 sec latency 1699.678 ms 2024-04-13T03:46:43.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:42 smithi016 ceph-mon[31772]: pgmap v373: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 192 op/s 2024-04-13T03:46:43.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:42 smithi188 ceph-mon[36398]: pgmap v373: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 192 op/s 2024-04-13T03:46:43.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:42 smithi119 ceph-mon[35835]: pgmap v373: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 192 op/s 2024-04-13T03:46:43.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 402563 3.41 MB/sec execute 504 sec latency 2670.598 ms 2024-04-13T03:46:44.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 403839 3.42 MB/sec execute 505 sec latency 54.481 ms 2024-04-13T03:46:45.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:44 smithi188 ceph-mon[36398]: pgmap v374: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-04-13T03:46:45.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:44 smithi119 ceph-mon[35835]: pgmap v374: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-04-13T03:46:45.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:44 smithi016 ceph-mon[31772]: pgmap v374: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-04-13T03:46:45.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 404764 3.43 MB/sec execute 506 sec latency 23.887 ms 2024-04-13T03:46:46.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 405390 3.42 MB/sec execute 507 sec latency 30.366 ms 2024-04-13T03:46:47.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:46 smithi188 ceph-mon[36398]: pgmap v375: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-04-13T03:46:47.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:46 smithi016 ceph-mon[31772]: pgmap v375: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-04-13T03:46:47.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:46 smithi119 ceph-mon[35835]: pgmap v375: 129 pgs: 129 active+clean; 346 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-04-13T03:46:47.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 406042 3.42 MB/sec execute 508 sec latency 30.009 ms 2024-04-13T03:46:48.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 406104 3.42 MB/sec execute 509 sec latency 902.311 ms 2024-04-13T03:46:49.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:48 smithi188 ceph-mon[36398]: pgmap v376: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 172 op/s 2024-04-13T03:46:49.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:48 smithi119 ceph-mon[35835]: pgmap v376: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 172 op/s 2024-04-13T03:46:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:48 smithi016 ceph-mon[31772]: pgmap v376: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 172 op/s 2024-04-13T03:46:49.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 407008 3.42 MB/sec execute 510 sec latency 21.697 ms 2024-04-13T03:46:50.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 408233 3.43 MB/sec execute 511 sec latency 47.054 ms 2024-04-13T03:46:51.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:50 smithi188 ceph-mon[36398]: pgmap v377: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 114 op/s 2024-04-13T03:46:51.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:50 smithi119 ceph-mon[35835]: pgmap v377: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 114 op/s 2024-04-13T03:46:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:50 smithi016 ceph-mon[31772]: pgmap v377: 129 pgs: 129 active+clean; 350 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 114 op/s 2024-04-13T03:46:51.950 INFO:tasks.workunit.client.0.smithi016.stdout: 1 408892 3.43 MB/sec execute 512 sec latency 33.676 ms 2024-04-13T03:46:52.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 409641 3.43 MB/sec execute 513 sec latency 313.748 ms 2024-04-13T03:46:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:52 smithi119 ceph-mon[35835]: pgmap v378: 129 pgs: 129 active+clean; 355 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 174 op/s 2024-04-13T03:46:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:52 smithi016 ceph-mon[31772]: pgmap v378: 129 pgs: 129 active+clean; 355 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 174 op/s 2024-04-13T03:46:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:52 smithi188 ceph-mon[36398]: pgmap v378: 129 pgs: 129 active+clean; 355 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 174 op/s 2024-04-13T03:46:53.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 409653 3.42 MB/sec execute 514 sec latency 1284.626 ms 2024-04-13T03:46:54.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 411155 3.43 MB/sec execute 515 sec latency 19.982 ms 2024-04-13T03:46:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:54 smithi119 ceph-mon[35835]: pgmap v379: 129 pgs: 129 active+clean; 355 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-13T03:46:55.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:54 smithi016 ceph-mon[31772]: pgmap v379: 129 pgs: 129 active+clean; 355 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-13T03:46:55.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:54 smithi188 ceph-mon[36398]: pgmap v379: 129 pgs: 129 active+clean; 355 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-13T03:46:55.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 411948 3.44 MB/sec execute 516 sec latency 42.985 ms 2024-04-13T03:46:56.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 412750 3.43 MB/sec execute 517 sec latency 24.482 ms 2024-04-13T03:46:57.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:56 smithi188 ceph-mon[36398]: pgmap v380: 129 pgs: 129 active+clean; 355 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-13T03:46:57.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:56 smithi119 ceph-mon[35835]: pgmap v380: 129 pgs: 129 active+clean; 355 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-13T03:46:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:56 smithi016 ceph-mon[31772]: pgmap v380: 129 pgs: 129 active+clean; 355 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-13T03:46:57.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 413183 3.43 MB/sec execute 518 sec latency 566.875 ms 2024-04-13T03:46:58.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 414101 3.43 MB/sec execute 519 sec latency 731.750 ms 2024-04-13T03:46:59.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:46:58 smithi119 ceph-mon[35835]: pgmap v381: 129 pgs: 129 active+clean; 360 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 167 op/s 2024-04-13T03:46:59.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:46:58 smithi016 ceph-mon[31772]: pgmap v381: 129 pgs: 129 active+clean; 360 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 167 op/s 2024-04-13T03:46:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:46:58 smithi188 ceph-mon[36398]: pgmap v381: 129 pgs: 129 active+clean; 360 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 167 op/s 2024-04-13T03:46:59.951 INFO:tasks.workunit.client.0.smithi016.stdout: 1 415325 3.45 MB/sec execute 520 sec latency 16.039 ms 2024-04-13T03:47:00.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 415954 3.44 MB/sec execute 521 sec latency 44.950 ms 2024-04-13T03:47:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:00 smithi119 ceph-mon[35835]: pgmap v382: 129 pgs: 129 active+clean; 360 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-04-13T03:47:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:00 smithi016 ceph-mon[31772]: pgmap v382: 129 pgs: 129 active+clean; 360 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-04-13T03:47:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:00 smithi188 ceph-mon[36398]: pgmap v382: 129 pgs: 129 active+clean; 360 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-04-13T03:47:01.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416073 3.44 MB/sec execute 522 sec latency 95.516 ms 2024-04-13T03:47:02.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416732 3.44 MB/sec execute 523 sec latency 105.302 ms 2024-04-13T03:47:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:02 smithi119 ceph-mon[35835]: pgmap v383: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 209 op/s 2024-04-13T03:47:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:02 smithi119 ceph-mon[35835]: osdmap e108: 12 total, 12 up, 12 in 2024-04-13T03:47:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:02 smithi016 ceph-mon[31772]: pgmap v383: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 209 op/s 2024-04-13T03:47:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:02 smithi016 ceph-mon[31772]: osdmap e108: 12 total, 12 up, 12 in 2024-04-13T03:47:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:02 smithi188 ceph-mon[36398]: pgmap v383: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 209 op/s 2024-04-13T03:47:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:02 smithi188 ceph-mon[36398]: osdmap e108: 12 total, 12 up, 12 in 2024-04-13T03:47:03.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 416744 3.43 MB/sec execute 524 sec latency 1044.273 ms 2024-04-13T03:47:04.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 417960 3.44 MB/sec execute 525 sec latency 21.772 ms 2024-04-13T03:47:05.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:04 smithi119 ceph-mon[35835]: pgmap v385: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-13T03:47:05.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:04 smithi016 ceph-mon[31772]: pgmap v385: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-13T03:47:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:04 smithi188 ceph-mon[36398]: pgmap v385: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-13T03:47:05.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 418955 3.44 MB/sec execute 526 sec latency 26.034 ms 2024-04-13T03:47:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 419642 3.44 MB/sec execute 527 sec latency 21.355 ms 2024-04-13T03:47:07.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:06 smithi119 ceph-mon[35835]: pgmap v386: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-13T03:47:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:06 smithi016 ceph-mon[31772]: pgmap v386: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-13T03:47:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:06 smithi188 ceph-mon[36398]: pgmap v386: 129 pgs: 129 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-13T03:47:07.952 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420275 3.44 MB/sec execute 528 sec latency 326.856 ms 2024-04-13T03:47:08.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 420287 3.43 MB/sec execute 529 sec latency 1297.125 ms 2024-04-13T03:47:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:08 smithi119 ceph-mon[35835]: pgmap v387: 129 pgs: 129 active+clean; 369 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-04-13T03:47:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:08 smithi119 ceph-mon[35835]: osdmap e109: 12 total, 12 up, 12 in 2024-04-13T03:47:09.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:08 smithi016 ceph-mon[31772]: pgmap v387: 129 pgs: 129 active+clean; 369 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-04-13T03:47:09.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:08 smithi016 ceph-mon[31772]: osdmap e109: 12 total, 12 up, 12 in 2024-04-13T03:47:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:08 smithi188 ceph-mon[36398]: pgmap v387: 129 pgs: 129 active+clean; 369 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-04-13T03:47:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:08 smithi188 ceph-mon[36398]: osdmap e109: 12 total, 12 up, 12 in 2024-04-13T03:47:09.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 421555 3.44 MB/sec execute 530 sec latency 21.798 ms 2024-04-13T03:47:10.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 422513 3.45 MB/sec execute 531 sec latency 26.928 ms 2024-04-13T03:47:11.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:10 smithi119 ceph-mon[35835]: pgmap v389: 129 pgs: 129 active+clean; 369 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-13T03:47:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:10 smithi016 ceph-mon[31772]: pgmap v389: 129 pgs: 129 active+clean; 369 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-13T03:47:11.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:10 smithi188 ceph-mon[36398]: pgmap v389: 129 pgs: 129 active+clean; 369 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-04-13T03:47:11.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 423223 3.44 MB/sec execute 532 sec latency 23.038 ms 2024-04-13T03:47:12.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 423824 3.44 MB/sec execute 533 sec latency 373.486 ms 2024-04-13T03:47:13.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:12 smithi119 ceph-mon[35835]: pgmap v390: 129 pgs: 129 active+clean; 374 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-04-13T03:47:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:12 smithi016 ceph-mon[31772]: pgmap v390: 129 pgs: 129 active+clean; 374 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-04-13T03:47:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:12 smithi188 ceph-mon[36398]: pgmap v390: 129 pgs: 129 active+clean; 374 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-04-13T03:47:13.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 423836 3.44 MB/sec execute 534 sec latency 1352.658 ms 2024-04-13T03:47:14.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 425298 3.45 MB/sec execute 535 sec latency 22.073 ms 2024-04-13T03:47:15.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:14 smithi119 ceph-mon[35835]: pgmap v391: 129 pgs: 129 active+clean; 374 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 153 op/s 2024-04-13T03:47:15.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:14 smithi016 ceph-mon[31772]: pgmap v391: 129 pgs: 129 active+clean; 374 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 153 op/s 2024-04-13T03:47:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:14 smithi188 ceph-mon[36398]: pgmap v391: 129 pgs: 129 active+clean; 374 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 153 op/s 2024-04-13T03:47:15.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 426154 3.45 MB/sec execute 536 sec latency 18.120 ms 2024-04-13T03:47:16.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427108 3.45 MB/sec execute 537 sec latency 27.685 ms 2024-04-13T03:47:17.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:16 smithi119 ceph-mon[35835]: pgmap v392: 129 pgs: 129 active+clean; 374 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 153 op/s 2024-04-13T03:47:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:16 smithi016 ceph-mon[31772]: pgmap v392: 129 pgs: 129 active+clean; 374 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 153 op/s 2024-04-13T03:47:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:16 smithi188 ceph-mon[36398]: pgmap v392: 129 pgs: 129 active+clean; 374 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 153 op/s 2024-04-13T03:47:17.953 INFO:tasks.workunit.client.0.smithi016.stdout: 1 427365 3.45 MB/sec execute 538 sec latency 732.541 ms 2024-04-13T03:47:18.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:17 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:47:18.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:17 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:47:18.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:17 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:47:18.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 428551 3.45 MB/sec execute 539 sec latency 894.647 ms 2024-04-13T03:47:19.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:18 smithi119 ceph-mon[35835]: pgmap v393: 129 pgs: 129 active+clean; 383 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 155 op/s 2024-04-13T03:47:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:18 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:47:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:18 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:47:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:18 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:47:19.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:18 smithi016 ceph-mon[31772]: pgmap v393: 129 pgs: 129 active+clean; 383 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 155 op/s 2024-04-13T03:47:19.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:18 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:47:19.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:18 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:47:19.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:18 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:47:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:18 smithi188 ceph-mon[36398]: pgmap v393: 129 pgs: 129 active+clean; 383 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 155 op/s 2024-04-13T03:47:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:18 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:47:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:18 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:47:19.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:18 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:47:19.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 429499 3.46 MB/sec execute 540 sec latency 22.141 ms 2024-04-13T03:47:20.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:19 smithi119 ceph-mon[35835]: pgmap v394: 129 pgs: 129 active+clean; 383 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 133 op/s 2024-04-13T03:47:20.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:19 smithi016 ceph-mon[31772]: pgmap v394: 129 pgs: 129 active+clean; 383 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 133 op/s 2024-04-13T03:47:20.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:19 smithi188 ceph-mon[36398]: pgmap v394: 129 pgs: 129 active+clean; 383 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 133 op/s 2024-04-13T03:47:20.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430012 3.45 MB/sec execute 541 sec latency 64.438 ms 2024-04-13T03:47:21.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430831 3.46 MB/sec execute 542 sec latency 100.622 ms 2024-04-13T03:47:22.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430914 3.45 MB/sec execute 543 sec latency 875.543 ms 2024-04-13T03:47:23.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:22 smithi016 ceph-mon[31772]: pgmap v395: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-04-13T03:47:23.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:22 smithi188 ceph-mon[36398]: pgmap v395: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-04-13T03:47:23.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:22 smithi119 ceph-mon[35835]: pgmap v395: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-04-13T03:47:23.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 430917 3.44 MB/sec execute 544 sec latency 1850.846 ms 2024-04-13T03:47:24.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 432121 3.45 MB/sec execute 545 sec latency 26.831 ms 2024-04-13T03:47:25.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:24 smithi016 ceph-mon[31772]: pgmap v396: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 132 op/s 2024-04-13T03:47:25.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:24 smithi188 ceph-mon[36398]: pgmap v396: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 132 op/s 2024-04-13T03:47:25.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:24 smithi119 ceph-mon[35835]: pgmap v396: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 132 op/s 2024-04-13T03:47:25.954 INFO:tasks.workunit.client.0.smithi016.stdout: 1 433088 3.46 MB/sec execute 546 sec latency 45.046 ms 2024-04-13T03:47:26.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 433639 3.45 MB/sec execute 547 sec latency 66.191 ms 2024-04-13T03:47:27.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:26 smithi188 ceph-mon[36398]: pgmap v397: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 132 op/s 2024-04-13T03:47:27.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:26 smithi119 ceph-mon[35835]: pgmap v397: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 132 op/s 2024-04-13T03:47:27.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:26 smithi016 ceph-mon[31772]: pgmap v397: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 132 op/s 2024-04-13T03:47:27.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434458 3.45 MB/sec execute 548 sec latency 100.402 ms 2024-04-13T03:47:28.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:27 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:28.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:27 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:28.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:27 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:28.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 434470 3.45 MB/sec execute 549 sec latency 1072.861 ms 2024-04-13T03:47:29.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:28 smithi119 ceph-mon[35835]: pgmap v398: 129 pgs: 129 active+clean; 393 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-04-13T03:47:29.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:28 smithi016 ceph-mon[31772]: pgmap v398: 129 pgs: 129 active+clean; 393 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-04-13T03:47:29.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:28 smithi188 ceph-mon[36398]: pgmap v398: 129 pgs: 129 active+clean; 393 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 182 op/s 2024-04-13T03:47:29.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 435729 3.45 MB/sec execute 550 sec latency 23.379 ms 2024-04-13T03:47:30.955 INFO:tasks.workunit.client.0.smithi016.stdout: 1 436708 3.46 MB/sec execute 551 sec latency 21.002 ms 2024-04-13T03:47:31.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:30 smithi119 ceph-mon[35835]: pgmap v399: 129 pgs: 129 active+clean; 393 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 113 op/s 2024-04-13T03:47:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:30 smithi016 ceph-mon[31772]: pgmap v399: 129 pgs: 129 active+clean; 393 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 113 op/s 2024-04-13T03:47:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:30 smithi188 ceph-mon[36398]: pgmap v399: 129 pgs: 129 active+clean; 393 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 113 op/s 2024-04-13T03:47:31.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 437492 3.46 MB/sec execute 552 sec latency 25.743 ms 2024-04-13T03:47:32.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438005 3.46 MB/sec execute 553 sec latency 495.886 ms 2024-04-13T03:47:33.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:32 smithi119 ceph-mon[35835]: pgmap v400: 129 pgs: 129 active+clean; 395 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 176 op/s 2024-04-13T03:47:33.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:32 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:33.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:32 smithi016 ceph-mon[31772]: pgmap v400: 129 pgs: 129 active+clean; 395 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 176 op/s 2024-04-13T03:47:33.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:32 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:32 smithi188 ceph-mon[36398]: pgmap v400: 129 pgs: 129 active+clean; 395 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 176 op/s 2024-04-13T03:47:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:32 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:33.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 438017 3.45 MB/sec execute 554 sec latency 1465.032 ms 2024-04-13T03:47:34.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 439480 3.46 MB/sec execute 555 sec latency 22.788 ms 2024-04-13T03:47:35.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:34 smithi119 ceph-mon[35835]: pgmap v401: 129 pgs: 129 active+clean; 395 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-04-13T03:47:35.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:34 smithi016 ceph-mon[31772]: pgmap v401: 129 pgs: 129 active+clean; 395 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-04-13T03:47:35.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:34 smithi188 ceph-mon[36398]: pgmap v401: 129 pgs: 129 active+clean; 395 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-04-13T03:47:35.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 440301 3.46 MB/sec execute 556 sec latency 17.958 ms 2024-04-13T03:47:36.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 441106 3.46 MB/sec execute 557 sec latency 20.184 ms 2024-04-13T03:47:37.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:36 smithi119 ceph-mon[35835]: pgmap v402: 129 pgs: 129 active+clean; 395 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-04-13T03:47:37.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:36 smithi016 ceph-mon[31772]: pgmap v402: 129 pgs: 129 active+clean; 395 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-04-13T03:47:37.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:36 smithi188 ceph-mon[36398]: pgmap v402: 129 pgs: 129 active+clean; 395 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 113 op/s 2024-04-13T03:47:37.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 441547 3.46 MB/sec execute 558 sec latency 583.853 ms 2024-04-13T03:47:38.956 INFO:tasks.workunit.client.0.smithi016.stdout: 1 442640 3.46 MB/sec execute 559 sec latency 745.465 ms 2024-04-13T03:47:39.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:38 smithi119 ceph-mon[35835]: pgmap v403: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-04-13T03:47:39.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:38 smithi016 ceph-mon[31772]: pgmap v403: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-04-13T03:47:39.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:38 smithi188 ceph-mon[36398]: pgmap v403: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 172 op/s 2024-04-13T03:47:39.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 443771 3.47 MB/sec execute 560 sec latency 16.310 ms 2024-04-13T03:47:40.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 444386 3.47 MB/sec execute 561 sec latency 23.635 ms 2024-04-13T03:47:41.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:40 smithi119 ceph-mon[35835]: pgmap v404: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-04-13T03:47:41.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:40 smithi016 ceph-mon[31772]: pgmap v404: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-04-13T03:47:41.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:40 smithi188 ceph-mon[36398]: pgmap v404: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 121 op/s 2024-04-13T03:47:41.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445096 3.47 MB/sec execute 562 sec latency 225.840 ms 2024-04-13T03:47:42.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445096 3.46 MB/sec execute 563 sec latency 1225.993 ms 2024-04-13T03:47:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:42 smithi119 ceph-mon[35835]: pgmap v405: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 188 op/s 2024-04-13T03:47:43.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:42 smithi016 ceph-mon[31772]: pgmap v405: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 188 op/s 2024-04-13T03:47:43.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:42 smithi188 ceph-mon[36398]: pgmap v405: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 188 op/s 2024-04-13T03:47:43.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 445099 3.46 MB/sec execute 564 sec latency 2207.365 ms 2024-04-13T03:47:44.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 446303 3.46 MB/sec execute 565 sec latency 27.930 ms 2024-04-13T03:47:45.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:44 smithi119 ceph-mon[35835]: pgmap v406: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-04-13T03:47:45.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:44 smithi016 ceph-mon[31772]: pgmap v406: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-04-13T03:47:45.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:44 smithi188 ceph-mon[36398]: pgmap v406: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-04-13T03:47:45.957 INFO:tasks.workunit.client.0.smithi016.stdout: 1 447202 3.47 MB/sec execute 566 sec latency 20.773 ms 2024-04-13T03:47:46.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 447701 3.46 MB/sec execute 567 sec latency 54.661 ms 2024-04-13T03:47:47.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:46 smithi119 ceph-mon[35835]: pgmap v407: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-04-13T03:47:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:46 smithi016 ceph-mon[31772]: pgmap v407: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-04-13T03:47:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:46 smithi188 ceph-mon[36398]: pgmap v407: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 125 op/s 2024-04-13T03:47:47.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 448465 3.47 MB/sec execute 568 sec latency 60.169 ms 2024-04-13T03:47:48.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:47 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:48.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:47 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:48.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:47 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:48.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 448650 3.46 MB/sec execute 569 sec latency 697.925 ms 2024-04-13T03:47:49.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:48 smithi119 ceph-mon[35835]: pgmap v408: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 171 op/s 2024-04-13T03:47:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:48 smithi016 ceph-mon[31772]: pgmap v408: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 171 op/s 2024-04-13T03:47:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:48 smithi188 ceph-mon[36398]: pgmap v408: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 171 op/s 2024-04-13T03:47:49.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 449875 3.47 MB/sec execute 570 sec latency 23.435 ms 2024-04-13T03:47:50.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:49 smithi119 ceph-mon[35835]: pgmap v409: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-04-13T03:47:50.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:49 smithi016 ceph-mon[31772]: pgmap v409: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-04-13T03:47:50.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:49 smithi188 ceph-mon[36398]: pgmap v409: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-04-13T03:47:50.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 450802 3.47 MB/sec execute 571 sec latency 22.003 ms 2024-04-13T03:47:51.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 451419 3.47 MB/sec execute 572 sec latency 41.102 ms 2024-04-13T03:47:52.958 INFO:tasks.workunit.client.0.smithi016.stdout: 1 452187 3.47 MB/sec execute 573 sec latency 178.090 ms 2024-04-13T03:47:53.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:52 smithi188 ceph-mon[36398]: pgmap v410: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 174 op/s 2024-04-13T03:47:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:52 smithi119 ceph-mon[35835]: pgmap v410: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 174 op/s 2024-04-13T03:47:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:52 smithi016 ceph-mon[31772]: pgmap v410: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 174 op/s 2024-04-13T03:47:53.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 452199 3.46 MB/sec execute 574 sec latency 1150.997 ms 2024-04-13T03:47:54.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 453394 3.47 MB/sec execute 575 sec latency 31.164 ms 2024-04-13T03:47:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:54 smithi188 ceph-mon[36398]: pgmap v411: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:47:55.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:54 smithi119 ceph-mon[35835]: pgmap v411: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:47:55.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:54 smithi016 ceph-mon[31772]: pgmap v411: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:47:55.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 454316 3.48 MB/sec execute 576 sec latency 18.435 ms 2024-04-13T03:47:56.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:55 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:56.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:55 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:56.375 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:55 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:47:56.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 454828 3.47 MB/sec execute 577 sec latency 20.563 ms 2024-04-13T03:47:57.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:56 smithi119 ceph-mon[35835]: pgmap v412: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:47:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:56 smithi016 ceph-mon[31772]: pgmap v412: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:47:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:56 smithi188 ceph-mon[36398]: pgmap v412: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 108 op/s 2024-04-13T03:47:57.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 455730 3.47 MB/sec execute 578 sec latency 50.007 ms 2024-04-13T03:47:58.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 456861 3.48 MB/sec execute 579 sec latency 209.785 ms 2024-04-13T03:47:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:47:58 smithi188 ceph-mon[36398]: pgmap v413: 129 pgs: 129 active+clean; 421 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 165 op/s 2024-04-13T03:47:59.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:47:58 smithi119 ceph-mon[35835]: pgmap v413: 129 pgs: 129 active+clean; 421 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 165 op/s 2024-04-13T03:47:59.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:47:58 smithi016 ceph-mon[31772]: pgmap v413: 129 pgs: 129 active+clean; 421 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 165 op/s 2024-04-13T03:47:59.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 457904 3.48 MB/sec execute 580 sec latency 15.936 ms 2024-04-13T03:48:00.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 458344 3.48 MB/sec execute 581 sec latency 134.962 ms 2024-04-13T03:48:01.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:00 smithi119 ceph-mon[35835]: pgmap v414: 129 pgs: 129 active+clean; 421 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-13T03:48:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:00 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:00 smithi016 ceph-mon[31772]: pgmap v414: 129 pgs: 129 active+clean; 421 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-13T03:48:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:00 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:00 smithi188 ceph-mon[36398]: pgmap v414: 129 pgs: 129 active+clean; 421 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-13T03:48:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:00 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:01.959 INFO:tasks.workunit.client.0.smithi016.stdout: 1 458797 3.50 MB/sec execute 582 sec latency 70.917 ms 2024-04-13T03:48:02.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 459762 3.51 MB/sec execute 583 sec latency 35.732 ms 2024-04-13T03:48:03.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:02 smithi119 ceph-mon[35835]: pgmap v415: 129 pgs: 129 active+clean; 426 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 200 op/s 2024-04-13T03:48:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:02 smithi119 ceph-mon[35835]: osdmap e110: 12 total, 12 up, 12 in 2024-04-13T03:48:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:02 smithi016 ceph-mon[31772]: pgmap v415: 129 pgs: 129 active+clean; 426 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 200 op/s 2024-04-13T03:48:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:02 smithi016 ceph-mon[31772]: osdmap e110: 12 total, 12 up, 12 in 2024-04-13T03:48:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:02 smithi188 ceph-mon[36398]: pgmap v415: 129 pgs: 129 active+clean; 426 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 200 op/s 2024-04-13T03:48:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:02 smithi188 ceph-mon[36398]: osdmap e110: 12 total, 12 up, 12 in 2024-04-13T03:48:03.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460042 3.51 MB/sec execute 584 sec latency 659.848 ms 2024-04-13T03:48:04.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460043 3.50 MB/sec execute 585 sec latency 980.360 ms 2024-04-13T03:48:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:04 smithi119 ceph-mon[35835]: pgmap v417: 129 pgs: 129 active+clean; 426 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-04-13T03:48:05.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:04 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:04 smithi016 ceph-mon[31772]: pgmap v417: 129 pgs: 129 active+clean; 426 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-04-13T03:48:05.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:04 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:04 smithi188 ceph-mon[36398]: pgmap v417: 129 pgs: 129 active+clean; 426 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-04-13T03:48:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:04 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:05.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460043 3.49 MB/sec execute 586 sec latency 1980.468 ms 2024-04-13T03:48:06.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460043 3.49 MB/sec execute 587 sec latency 2980.635 ms 2024-04-13T03:48:07.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:06 smithi119 ceph-mon[35835]: pgmap v418: 129 pgs: 129 active+clean; 426 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-04-13T03:48:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:06 smithi016 ceph-mon[31772]: pgmap v418: 129 pgs: 129 active+clean; 426 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-04-13T03:48:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:06 smithi188 ceph-mon[36398]: pgmap v418: 129 pgs: 129 active+clean; 426 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-04-13T03:48:07.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 460043 3.48 MB/sec execute 588 sec latency 3980.779 ms 2024-04-13T03:48:08.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 461123 3.49 MB/sec execute 589 sec latency 4133.681 ms 2024-04-13T03:48:09.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:08 smithi119 ceph-mon[35835]: pgmap v419: 129 pgs: 129 active+clean; 425 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-04-13T03:48:09.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:08 smithi119 ceph-mon[35835]: osdmap e111: 12 total, 12 up, 12 in 2024-04-13T03:48:09.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:08 smithi016 ceph-mon[31772]: pgmap v419: 129 pgs: 129 active+clean; 425 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-04-13T03:48:09.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:08 smithi016 ceph-mon[31772]: osdmap e111: 12 total, 12 up, 12 in 2024-04-13T03:48:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:08 smithi188 ceph-mon[36398]: pgmap v419: 129 pgs: 129 active+clean; 425 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-04-13T03:48:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:08 smithi188 ceph-mon[36398]: osdmap e111: 12 total, 12 up, 12 in 2024-04-13T03:48:09.960 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462145 3.49 MB/sec execute 590 sec latency 26.012 ms 2024-04-13T03:48:10.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462499 3.49 MB/sec execute 591 sec latency 370.798 ms 2024-04-13T03:48:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:10 smithi119 ceph-mon[35835]: pgmap v421: 129 pgs: 129 active+clean; 425 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 789 KiB/s wr, 63 op/s 2024-04-13T03:48:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:10 smithi016 ceph-mon[31772]: pgmap v421: 129 pgs: 129 active+clean; 425 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 789 KiB/s wr, 63 op/s 2024-04-13T03:48:11.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:10 smithi188 ceph-mon[36398]: pgmap v421: 129 pgs: 129 active+clean; 425 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 789 KiB/s wr, 63 op/s 2024-04-13T03:48:11.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462499 3.48 MB/sec execute 592 sec latency 1370.892 ms 2024-04-13T03:48:12.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462499 3.48 MB/sec execute 593 sec latency 2371.020 ms 2024-04-13T03:48:13.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:12 smithi119 ceph-mon[35835]: pgmap v422: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.5 MiB/s wr, 104 op/s 2024-04-13T03:48:13.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:12 smithi016 ceph-mon[31772]: pgmap v422: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.5 MiB/s wr, 104 op/s 2024-04-13T03:48:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:12 smithi188 ceph-mon[36398]: pgmap v422: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.5 MiB/s wr, 104 op/s 2024-04-13T03:48:13.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462499 3.47 MB/sec execute 594 sec latency 3371.142 ms 2024-04-13T03:48:14.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462499 3.46 MB/sec execute 595 sec latency 4371.278 ms 2024-04-13T03:48:15.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:14 smithi119 ceph-mon[35835]: pgmap v423: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-13T03:48:15.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:14 smithi016 ceph-mon[31772]: pgmap v423: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-13T03:48:15.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:14 smithi188 ceph-mon[36398]: pgmap v423: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-13T03:48:15.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462499 3.46 MB/sec execute 596 sec latency 5371.423 ms 2024-04-13T03:48:16.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462499 3.45 MB/sec execute 597 sec latency 6371.594 ms 2024-04-13T03:48:17.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:16 smithi119 ceph-mon[35835]: pgmap v424: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-13T03:48:17.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:16 smithi016 ceph-mon[31772]: pgmap v424: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-13T03:48:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:16 smithi188 ceph-mon[36398]: pgmap v424: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-13T03:48:17.961 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462499 3.45 MB/sec execute 598 sec latency 7371.736 ms 2024-04-13T03:48:18.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:17 smithi119 ceph-mon[35835]: pgmap v425: 129 pgs: 129 active+clean; 432 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-04-13T03:48:18.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:17 smithi016 ceph-mon[31772]: pgmap v425: 129 pgs: 129 active+clean; 432 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-04-13T03:48:18.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:17 smithi188 ceph-mon[36398]: pgmap v425: 129 pgs: 129 active+clean; 432 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-04-13T03:48:18.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 462570 3.44 MB/sec execute 599 sec latency 8333.796 ms 2024-04-13T03:48:19.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:18 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:48:19.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:18 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:48:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:18 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:48:19.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 600 sec 2024-04-13T03:48:20.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:19 smithi119 ceph-mon[35835]: pgmap v426: 129 pgs: 129 active+clean; 432 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.3 MiB/s wr, 72 op/s 2024-04-13T03:48:20.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:19 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:48:20.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:19 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:48:20.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:19 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:48:20.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:19 smithi016 ceph-mon[31772]: pgmap v426: 129 pgs: 129 active+clean; 432 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.3 MiB/s wr, 72 op/s 2024-04-13T03:48:20.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:19 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:48:20.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:19 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:48:20.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:19 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:48:20.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:19 smithi188 ceph-mon[36398]: pgmap v426: 129 pgs: 129 active+clean; 432 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.3 MiB/s wr, 72 op/s 2024-04-13T03:48:20.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:19 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:48:20.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:19 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:48:20.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:19 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:48:20.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 601 sec 2024-04-13T03:48:21.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 602 sec 2024-04-13T03:48:22.963 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 603 sec 2024-04-13T03:48:23.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:22 smithi119 ceph-mon[35835]: pgmap v427: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.7 MiB/s wr, 100 op/s 2024-04-13T03:48:23.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:22 smithi016 ceph-mon[31772]: pgmap v427: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.7 MiB/s wr, 100 op/s 2024-04-13T03:48:23.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:22 smithi188 ceph-mon[36398]: pgmap v427: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.7 MiB/s wr, 100 op/s 2024-04-13T03:48:23.962 INFO:tasks.workunit.client.0.smithi016.stdout: 1 cleanup 604 sec 2024-04-13T03:48:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: 0 cleanup 604 sec 2024-04-13T03:48:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-04-13T03:48:24.170 INFO:tasks.workunit.client.0.smithi016.stdout: Operation Count AvgLat MaxLat 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: ---------------------------------------- 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: NTCreateX 65680 4.172 12168.105 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Close 48251 0.012 0.688 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Rename 2783 34.063 3997.349 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Unlink 13270 3.967 9801.499 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Deltree 2 155.113 239.327 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Mkdir 1 0.046 0.046 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Qpathinfo 59548 1.853 4804.299 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Qfileinfo 10446 0.005 0.142 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Qfsinfo 10924 0.547 47.419 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Sfileinfo 5354 0.088 7.467 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: Find 23031 0.470 64.418 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: WriteX 32830 0.358 152.462 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: ReadX 102980 0.057 47.796 2024-04-13T03:48:24.171 INFO:tasks.workunit.client.0.smithi016.stdout: LockX 214 0.943 2.919 2024-04-13T03:48:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: UnlockX 214 0.821 3.246 2024-04-13T03:48:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: Flush 4607 7.092 57.815 2024-04-13T03:48:24.172 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-04-13T03:48:24.172 INFO:tasks.workunit.client.0.smithi016.stdout:Throughput 3.44196 MB/sec 1 clients 1 procs max_latency=12168.121 ms 2024-04-13T03:48:24.172 INFO:tasks.workunit.client.0.smithi016.stderr:+ dbench 10 2024-04-13T03:48:24.174 INFO:tasks.workunit.client.0.smithi016.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-13T03:48:24.175 INFO:tasks.workunit.client.0.smithi016.stdout: 2024-04-13T03:48:24.175 INFO:tasks.workunit.client.0.smithi016.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-13T03:48:24.178 INFO:tasks.workunit.client.0.smithi016.stdout:8 of 10 processes prepared for launch 0 sec 2024-04-13T03:48:24.279 INFO:tasks.workunit.client.0.smithi016.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-13T03:48:24.279 INFO:tasks.workunit.client.0.smithi016.stdout:releasing clients 2024-04-13T03:48:25.279 INFO:tasks.workunit.client.0.smithi016.stdout: 10 332 120.24 MB/sec warmup 1 sec latency 110.964 ms 2024-04-13T03:48:25.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:24 smithi119 ceph-mon[35835]: pgmap v428: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1003 KiB/s wr, 56 op/s 2024-04-13T03:48:25.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:24 smithi016 ceph-mon[31772]: pgmap v428: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1003 KiB/s wr, 56 op/s 2024-04-13T03:48:25.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:24 smithi188 ceph-mon[36398]: pgmap v428: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1003 KiB/s wr, 56 op/s 2024-04-13T03:48:26.280 INFO:tasks.workunit.client.0.smithi016.stdout: 10 690 102.75 MB/sec warmup 2 sec latency 99.724 ms 2024-04-13T03:48:27.280 INFO:tasks.workunit.client.0.smithi016.stdout: 10 1044 74.31 MB/sec warmup 3 sec latency 150.133 ms 2024-04-13T03:48:27.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:26 smithi119 ceph-mon[35835]: pgmap v429: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1003 KiB/s wr, 56 op/s 2024-04-13T03:48:27.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:26 smithi016 ceph-mon[31772]: pgmap v429: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1003 KiB/s wr, 56 op/s 2024-04-13T03:48:27.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:26 smithi188 ceph-mon[36398]: pgmap v429: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 1003 KiB/s wr, 56 op/s 2024-04-13T03:48:28.280 INFO:tasks.workunit.client.0.smithi016.stdout: 10 1489 64.10 MB/sec warmup 4 sec latency 124.573 ms 2024-04-13T03:48:28.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:27 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:28.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:27 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:28.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:27 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:29.272 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:28 smithi188 ceph-mon[36398]: pgmap v430: 129 pgs: 129 active+clean; 439 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.8 MiB/s wr, 114 op/s 2024-04-13T03:48:29.280 INFO:tasks.workunit.client.0.smithi016.stdout: 10 2244 54.88 MB/sec warmup 5 sec latency 97.518 ms 2024-04-13T03:48:29.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:28 smithi119 ceph-mon[35835]: pgmap v430: 129 pgs: 129 active+clean; 439 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.8 MiB/s wr, 114 op/s 2024-04-13T03:48:29.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:28 smithi016 ceph-mon[31772]: pgmap v430: 129 pgs: 129 active+clean; 439 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 1.8 MiB/s wr, 114 op/s 2024-04-13T03:48:30.280 INFO:tasks.workunit.client.0.smithi016.stdout: 10 2712 52.90 MB/sec warmup 6 sec latency 83.033 ms 2024-04-13T03:48:30.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:29 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:30.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:29 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:30.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:29 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:31.280 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3208 49.71 MB/sec warmup 7 sec latency 132.743 ms 2024-04-13T03:48:31.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:30 smithi119 ceph-mon[35835]: pgmap v431: 129 pgs: 129 active+clean; 439 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.3 MiB/s wr, 88 op/s 2024-04-13T03:48:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:30 smithi016 ceph-mon[31772]: pgmap v431: 129 pgs: 129 active+clean; 439 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.3 MiB/s wr, 88 op/s 2024-04-13T03:48:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:30 smithi188 ceph-mon[36398]: pgmap v431: 129 pgs: 129 active+clean; 439 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.3 MiB/s wr, 88 op/s 2024-04-13T03:48:32.281 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3624 48.07 MB/sec warmup 8 sec latency 133.984 ms 2024-04-13T03:48:33.281 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3767 43.08 MB/sec warmup 9 sec latency 85.465 ms 2024-04-13T03:48:33.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:32 smithi119 ceph-mon[35835]: pgmap v432: 129 pgs: 129 active+clean; 457 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 179 op/s 2024-04-13T03:48:33.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:32 smithi016 ceph-mon[31772]: pgmap v432: 129 pgs: 129 active+clean; 457 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 179 op/s 2024-04-13T03:48:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:32 smithi188 ceph-mon[36398]: pgmap v432: 129 pgs: 129 active+clean; 457 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 179 op/s 2024-04-13T03:48:34.281 INFO:tasks.workunit.client.0.smithi016.stdout: 10 3917 39.26 MB/sec warmup 10 sec latency 82.527 ms 2024-04-13T03:48:35.281 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4080 35.92 MB/sec warmup 11 sec latency 103.966 ms 2024-04-13T03:48:35.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:34 smithi119 ceph-mon[35835]: pgmap v433: 129 pgs: 129 active+clean; 457 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-04-13T03:48:35.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:34 smithi016 ceph-mon[31772]: pgmap v433: 129 pgs: 129 active+clean; 457 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-04-13T03:48:35.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:34 smithi188 ceph-mon[36398]: pgmap v433: 129 pgs: 129 active+clean; 457 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-04-13T03:48:36.281 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4421 33.80 MB/sec warmup 12 sec latency 70.318 ms 2024-04-13T03:48:36.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:35 smithi119 ceph-mon[35835]: pgmap v434: 129 pgs: 129 active+clean; 457 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-04-13T03:48:36.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:35 smithi016 ceph-mon[31772]: pgmap v434: 129 pgs: 129 active+clean; 457 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-04-13T03:48:36.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:35 smithi188 ceph-mon[36398]: pgmap v434: 129 pgs: 129 active+clean; 457 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 150 op/s 2024-04-13T03:48:37.281 INFO:tasks.workunit.client.0.smithi016.stdout: 10 4860 33.80 MB/sec warmup 13 sec latency 59.403 ms 2024-04-13T03:48:38.281 INFO:tasks.workunit.client.0.smithi016.stdout: 10 5314 32.20 MB/sec warmup 14 sec latency 90.530 ms 2024-04-13T03:48:39.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 5854 31.41 MB/sec warmup 15 sec latency 50.831 ms 2024-04-13T03:48:39.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:38 smithi119 ceph-mon[35835]: pgmap v435: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 238 op/s 2024-04-13T03:48:39.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:38 smithi016 ceph-mon[31772]: pgmap v435: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 238 op/s 2024-04-13T03:48:39.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:38 smithi188 ceph-mon[36398]: pgmap v435: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 238 op/s 2024-04-13T03:48:40.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6308 32.11 MB/sec warmup 16 sec latency 53.522 ms 2024-04-13T03:48:41.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 6770 31.91 MB/sec warmup 17 sec latency 62.493 ms 2024-04-13T03:48:41.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:40 smithi119 ceph-mon[35835]: pgmap v436: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 MiB/s wr, 179 op/s 2024-04-13T03:48:41.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:40 smithi016 ceph-mon[31772]: pgmap v436: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 MiB/s wr, 179 op/s 2024-04-13T03:48:41.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:40 smithi188 ceph-mon[36398]: pgmap v436: 129 pgs: 129 active+clean; 472 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.0 MiB/s wr, 179 op/s 2024-04-13T03:48:42.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7098 31.80 MB/sec warmup 18 sec latency 121.774 ms 2024-04-13T03:48:43.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7194 30.30 MB/sec warmup 19 sec latency 497.587 ms 2024-04-13T03:48:43.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:42 smithi119 ceph-mon[35835]: pgmap v437: 129 pgs: 129 active+clean; 500 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.8 MiB/s wr, 265 op/s 2024-04-13T03:48:43.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:42 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:43.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:42 smithi016 ceph-mon[31772]: pgmap v437: 129 pgs: 129 active+clean; 500 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.8 MiB/s wr, 265 op/s 2024-04-13T03:48:43.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:42 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:43.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:42 smithi188 ceph-mon[36398]: pgmap v437: 129 pgs: 129 active+clean; 500 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.8 MiB/s wr, 265 op/s 2024-04-13T03:48:43.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:42 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:44.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7252 28.82 MB/sec warmup 20 sec latency 756.477 ms 2024-04-13T03:48:45.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7510 27.87 MB/sec warmup 21 sec latency 338.919 ms 2024-04-13T03:48:45.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:44 smithi119 ceph-mon[35835]: pgmap v438: 129 pgs: 129 active+clean; 500 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 173 op/s 2024-04-13T03:48:45.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:44 smithi016 ceph-mon[31772]: pgmap v438: 129 pgs: 129 active+clean; 500 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 173 op/s 2024-04-13T03:48:45.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:44 smithi188 ceph-mon[36398]: pgmap v438: 129 pgs: 129 active+clean; 500 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 173 op/s 2024-04-13T03:48:46.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 7863 27.13 MB/sec warmup 22 sec latency 149.862 ms 2024-04-13T03:48:47.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 8237 26.82 MB/sec warmup 23 sec latency 293.842 ms 2024-04-13T03:48:47.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:46 smithi119 ceph-mon[35835]: pgmap v439: 129 pgs: 129 active+clean; 500 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 173 op/s 2024-04-13T03:48:47.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:46 smithi016 ceph-mon[31772]: pgmap v439: 129 pgs: 129 active+clean; 500 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 173 op/s 2024-04-13T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:46 smithi188 ceph-mon[36398]: pgmap v439: 129 pgs: 129 active+clean; 500 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.9 MiB/s wr, 173 op/s 2024-04-13T03:48:48.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 9079 27.47 MB/sec warmup 24 sec latency 39.177 ms 2024-04-13T03:48:48.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:47 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:48.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:47 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:48.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:47 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:48:49.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10160 28.79 MB/sec warmup 25 sec latency 56.820 ms 2024-04-13T03:48:49.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:48 smithi119 ceph-mon[35835]: pgmap v440: 129 pgs: 129 active+clean; 690 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 25 MiB/s wr, 388 op/s 2024-04-13T03:48:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:48 smithi016 ceph-mon[31772]: pgmap v440: 129 pgs: 129 active+clean; 690 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 25 MiB/s wr, 388 op/s 2024-04-13T03:48:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:48 smithi188 ceph-mon[36398]: pgmap v440: 129 pgs: 129 active+clean; 690 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 25 MiB/s wr, 388 op/s 2024-04-13T03:48:50.282 INFO:tasks.workunit.client.0.smithi016.stdout: 10 10970 29.78 MB/sec warmup 26 sec latency 45.510 ms 2024-04-13T03:48:51.283 INFO:tasks.workunit.client.0.smithi016.stdout: 10 11447 29.50 MB/sec warmup 27 sec latency 207.220 ms 2024-04-13T03:48:51.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:50 smithi119 ceph-mon[35835]: pgmap v441: 129 pgs: 129 active+clean; 690 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 22 MiB/s wr, 300 op/s 2024-04-13T03:48:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:50 smithi016 ceph-mon[31772]: pgmap v441: 129 pgs: 129 active+clean; 690 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 22 MiB/s wr, 300 op/s 2024-04-13T03:48:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:50 smithi188 ceph-mon[36398]: pgmap v441: 129 pgs: 129 active+clean; 690 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 22 MiB/s wr, 300 op/s 2024-04-13T03:48:52.283 INFO:tasks.workunit.client.0.smithi016.stdout: 10 11839 28.99 MB/sec warmup 28 sec latency 112.630 ms 2024-04-13T03:48:53.283 INFO:tasks.workunit.client.0.smithi016.stdout: 10 12790 29.35 MB/sec warmup 29 sec latency 114.971 ms 2024-04-13T03:48:53.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:52 smithi119 ceph-mon[35835]: pgmap v442: 129 pgs: 129 active+clean; 715 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 24 MiB/s wr, 499 op/s 2024-04-13T03:48:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:52 smithi016 ceph-mon[31772]: pgmap v442: 129 pgs: 129 active+clean; 715 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 24 MiB/s wr, 499 op/s 2024-04-13T03:48:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:52 smithi188 ceph-mon[36398]: pgmap v442: 129 pgs: 129 active+clean; 715 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 24 MiB/s wr, 499 op/s 2024-04-13T03:48:54.283 INFO:tasks.workunit.client.0.smithi016.stdout: 10 13854 30.63 MB/sec warmup 30 sec latency 52.038 ms 2024-04-13T03:48:55.283 INFO:tasks.workunit.client.0.smithi016.stdout: 10 14592 31.18 MB/sec warmup 31 sec latency 34.105 ms 2024-04-13T03:48:55.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:54 smithi119 ceph-mon[35835]: pgmap v443: 129 pgs: 129 active+clean; 715 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 21 MiB/s wr, 413 op/s 2024-04-13T03:48:55.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:54 smithi016 ceph-mon[31772]: pgmap v443: 129 pgs: 129 active+clean; 715 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 21 MiB/s wr, 413 op/s 2024-04-13T03:48:55.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:54 smithi188 ceph-mon[36398]: pgmap v443: 129 pgs: 129 active+clean; 715 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 21 MiB/s wr, 413 op/s 2024-04-13T03:48:56.283 INFO:tasks.workunit.client.0.smithi016.stdout: 10 15168 31.16 MB/sec warmup 32 sec latency 119.325 ms 2024-04-13T03:48:57.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 15781 31.01 MB/sec warmup 33 sec latency 124.240 ms 2024-04-13T03:48:57.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:56 smithi119 ceph-mon[35835]: pgmap v444: 129 pgs: 129 active+clean; 715 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 21 MiB/s wr, 413 op/s 2024-04-13T03:48:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:56 smithi016 ceph-mon[31772]: pgmap v444: 129 pgs: 129 active+clean; 715 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 21 MiB/s wr, 413 op/s 2024-04-13T03:48:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:56 smithi188 ceph-mon[36398]: pgmap v444: 129 pgs: 129 active+clean; 715 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 21 MiB/s wr, 413 op/s 2024-04-13T03:48:58.283 INFO:tasks.workunit.client.0.smithi016.stdout: 10 16729 31.57 MB/sec warmup 34 sec latency 40.158 ms 2024-04-13T03:48:59.283 INFO:tasks.workunit.client.0.smithi016.stdout: 10 17577 32.21 MB/sec warmup 35 sec latency 35.921 ms 2024-04-13T03:48:59.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:48:58 smithi119 ceph-mon[35835]: pgmap v445: 129 pgs: 129 active+clean; 740 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 24 MiB/s wr, 633 op/s 2024-04-13T03:48:59.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:48:58 smithi016 ceph-mon[31772]: pgmap v445: 129 pgs: 129 active+clean; 740 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 24 MiB/s wr, 633 op/s 2024-04-13T03:48:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:48:58 smithi188 ceph-mon[36398]: pgmap v445: 129 pgs: 129 active+clean; 740 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 24 MiB/s wr, 633 op/s 2024-04-13T03:49:00.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 18153 32.16 MB/sec warmup 36 sec latency 91.600 ms 2024-04-13T03:49:01.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 18312 31.37 MB/sec warmup 37 sec latency 743.080 ms 2024-04-13T03:49:01.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:00 smithi119 ceph-mon[35835]: pgmap v446: 129 pgs: 129 active+clean; 740 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 419 op/s 2024-04-13T03:49:01.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:00 smithi016 ceph-mon[31772]: pgmap v446: 129 pgs: 129 active+clean; 740 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 419 op/s 2024-04-13T03:49:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:00 smithi188 ceph-mon[36398]: pgmap v446: 129 pgs: 129 active+clean; 740 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 419 op/s 2024-04-13T03:49:02.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 18584 30.72 MB/sec warmup 38 sec latency 324.028 ms 2024-04-13T03:49:03.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 18599 30.05 MB/sec warmup 39 sec latency 1324.081 ms 2024-04-13T03:49:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:02 smithi119 ceph-mon[35835]: pgmap v447: 129 pgs: 129 active+clean; 761 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 614 op/s 2024-04-13T03:49:03.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:02 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:49:03.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:02 smithi016 ceph-mon[31772]: pgmap v447: 129 pgs: 129 active+clean; 761 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 614 op/s 2024-04-13T03:49:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:02 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:49:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:02 smithi188 ceph-mon[36398]: pgmap v447: 129 pgs: 129 active+clean; 761 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 614 op/s 2024-04-13T03:49:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:02 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:49:04.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 18862 29.70 MB/sec warmup 40 sec latency 2324.157 ms 2024-04-13T03:49:05.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 19759 30.12 MB/sec warmup 41 sec latency 3324.231 ms 2024-04-13T03:49:05.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:04 smithi119 ceph-mon[35835]: pgmap v448: 129 pgs: 129 active+clean; 761 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.7 MiB/s wr, 415 op/s 2024-04-13T03:49:05.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:04 smithi016 ceph-mon[31772]: pgmap v448: 129 pgs: 129 active+clean; 761 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.7 MiB/s wr, 415 op/s 2024-04-13T03:49:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:04 smithi188 ceph-mon[36398]: pgmap v448: 129 pgs: 129 active+clean; 761 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.7 MiB/s wr, 415 op/s 2024-04-13T03:49:06.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 20480 30.58 MB/sec warmup 42 sec latency 4324.303 ms 2024-04-13T03:49:07.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 20854 30.36 MB/sec warmup 43 sec latency 5324.394 ms 2024-04-13T03:49:07.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:06 smithi119 ceph-mon[35835]: pgmap v449: 129 pgs: 129 active+clean; 761 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.7 MiB/s wr, 415 op/s 2024-04-13T03:49:07.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:06 smithi016 ceph-mon[31772]: pgmap v449: 129 pgs: 129 active+clean; 761 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.7 MiB/s wr, 415 op/s 2024-04-13T03:49:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:06 smithi188 ceph-mon[36398]: pgmap v449: 129 pgs: 129 active+clean; 761 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.7 MiB/s wr, 415 op/s 2024-04-13T03:49:08.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 21298 29.91 MB/sec warmup 44 sec latency 6324.461 ms 2024-04-13T03:49:08.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:07 smithi119 ceph-mon[35835]: osdmap e112: 12 total, 12 up, 12 in 2024-04-13T03:49:08.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:07 smithi119 ceph-mon[35835]: pgmap v451: 129 pgs: 129 active+clean; 776 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 5.1 MiB/s wr, 589 op/s 2024-04-13T03:49:08.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:07 smithi016 ceph-mon[31772]: osdmap e112: 12 total, 12 up, 12 in 2024-04-13T03:49:08.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:07 smithi016 ceph-mon[31772]: pgmap v451: 129 pgs: 129 active+clean; 776 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 5.1 MiB/s wr, 589 op/s 2024-04-13T03:49:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:07 smithi188 ceph-mon[36398]: osdmap e112: 12 total, 12 up, 12 in 2024-04-13T03:49:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:07 smithi188 ceph-mon[36398]: pgmap v451: 129 pgs: 129 active+clean; 776 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 5.1 MiB/s wr, 589 op/s 2024-04-13T03:49:09.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 22049 30.12 MB/sec warmup 45 sec latency 7324.514 ms 2024-04-13T03:49:10.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 22919 30.56 MB/sec warmup 46 sec latency 8324.589 ms 2024-04-13T03:49:11.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 23614 30.88 MB/sec warmup 47 sec latency 9324.675 ms 2024-04-13T03:49:11.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:10 smithi119 ceph-mon[35835]: pgmap v452: 129 pgs: 129 active+clean; 776 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 5.1 MiB/s wr, 589 op/s 2024-04-13T03:49:11.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:10 smithi016 ceph-mon[31772]: pgmap v452: 129 pgs: 129 active+clean; 776 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 5.1 MiB/s wr, 589 op/s 2024-04-13T03:49:11.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:10 smithi188 ceph-mon[36398]: pgmap v452: 129 pgs: 129 active+clean; 776 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 5.1 MiB/s wr, 589 op/s 2024-04-13T03:49:12.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 24120 30.77 MB/sec warmup 48 sec latency 10324.757 ms 2024-04-13T03:49:13.284 INFO:tasks.workunit.client.0.smithi016.stdout: 10 24693 30.66 MB/sec warmup 49 sec latency 11324.830 ms 2024-04-13T03:49:13.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:12 smithi119 ceph-mon[35835]: pgmap v453: 129 pgs: 129 active+clean; 801 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 624 op/s 2024-04-13T03:49:13.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:12 smithi016 ceph-mon[31772]: pgmap v453: 129 pgs: 129 active+clean; 801 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 624 op/s 2024-04-13T03:49:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:12 smithi188 ceph-mon[36398]: pgmap v453: 129 pgs: 129 active+clean; 801 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 624 op/s 2024-04-13T03:49:14.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 25482 30.86 MB/sec warmup 50 sec latency 12324.895 ms 2024-04-13T03:49:14.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:14 smithi119 ceph-mon[35835]: osdmap e113: 12 total, 12 up, 12 in 2024-04-13T03:49:14.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:14 smithi119 ceph-mon[35835]: pgmap v455: 129 pgs: 129 active+clean; 801 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.4 MiB/s wr, 781 op/s 2024-04-13T03:49:14.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:14 smithi016 ceph-mon[31772]: osdmap e113: 12 total, 12 up, 12 in 2024-04-13T03:49:14.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:14 smithi016 ceph-mon[31772]: pgmap v455: 129 pgs: 129 active+clean; 801 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.4 MiB/s wr, 781 op/s 2024-04-13T03:49:14.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:14 smithi188 ceph-mon[36398]: osdmap e113: 12 total, 12 up, 12 in 2024-04-13T03:49:14.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:14 smithi188 ceph-mon[36398]: pgmap v455: 129 pgs: 129 active+clean; 801 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.4 MiB/s wr, 781 op/s 2024-04-13T03:49:15.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 26026 30.76 MB/sec warmup 51 sec latency 13324.998 ms 2024-04-13T03:49:16.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 26716 31.27 MB/sec warmup 52 sec latency 14325.068 ms 2024-04-13T03:49:17.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 27139 30.90 MB/sec warmup 53 sec latency 14936.382 ms 2024-04-13T03:49:17.307 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:16 smithi016 ceph-mon[31772]: pgmap v456: 129 pgs: 129 active+clean; 801 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.9 MiB/s wr, 305 op/s 2024-04-13T03:49:17.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:16 smithi119 ceph-mon[35835]: pgmap v456: 129 pgs: 129 active+clean; 801 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.9 MiB/s wr, 305 op/s 2024-04-13T03:49:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:16 smithi188 ceph-mon[36398]: pgmap v456: 129 pgs: 129 active+clean; 801 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.9 MiB/s wr, 305 op/s 2024-04-13T03:49:18.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 27746 30.91 MB/sec warmup 54 sec latency 2035.298 ms 2024-04-13T03:49:19.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 28661 31.13 MB/sec warmup 55 sec latency 3035.365 ms 2024-04-13T03:49:19.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:18 smithi119 ceph-mon[35835]: pgmap v457: 129 pgs: 129 active+clean; 831 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 540 op/s 2024-04-13T03:49:19.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:18 smithi016 ceph-mon[31772]: pgmap v457: 129 pgs: 129 active+clean; 831 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 540 op/s 2024-04-13T03:49:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:18 smithi188 ceph-mon[36398]: pgmap v457: 129 pgs: 129 active+clean; 831 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 540 op/s 2024-04-13T03:49:20.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 29135 31.08 MB/sec warmup 56 sec latency 3954.916 ms 2024-04-13T03:49:21.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 29610 31.10 MB/sec warmup 57 sec latency 93.304 ms 2024-04-13T03:49:21.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:20 smithi119 ceph-mon[35835]: pgmap v458: 129 pgs: 129 active+clean; 831 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 540 op/s 2024-04-13T03:49:21.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:20 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:49:21.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:20 smithi016 ceph-mon[31772]: pgmap v458: 129 pgs: 129 active+clean; 831 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 540 op/s 2024-04-13T03:49:21.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:20 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:49:21.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:20 smithi188 ceph-mon[36398]: pgmap v458: 129 pgs: 129 active+clean; 831 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 540 op/s 2024-04-13T03:49:21.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:20 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:49:22.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 30205 31.10 MB/sec warmup 58 sec latency 49.411 ms 2024-04-13T03:49:22.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:21 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:49:22.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:21 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:49:22.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:21 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:49:22.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:21 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:49:22.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:21 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:49:22.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:21 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:49:22.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:21 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:49:22.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:21 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:49:22.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:21 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:49:23.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 30870 31.14 MB/sec warmup 59 sec latency 85.299 ms 2024-04-13T03:49:23.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:22 smithi119 ceph-mon[35835]: pgmap v459: 129 pgs: 129 active+clean; 864 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 644 op/s 2024-04-13T03:49:23.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:22 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:49:23.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:22 smithi016 ceph-mon[31772]: pgmap v459: 129 pgs: 129 active+clean; 864 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 644 op/s 2024-04-13T03:49:23.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:22 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:49:23.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:22 smithi188 ceph-mon[36398]: pgmap v459: 129 pgs: 129 active+clean; 864 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 644 op/s 2024-04-13T03:49:23.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:22 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:49:24.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 31785 31.26 MB/sec warmup 60 sec latency 56.607 ms 2024-04-13T03:49:24.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:24 smithi119 ceph-mon[35835]: pgmap v460: 129 pgs: 129 active+clean; 864 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 606 op/s 2024-04-13T03:49:24.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:24 smithi016 ceph-mon[31772]: pgmap v460: 129 pgs: 129 active+clean; 864 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 606 op/s 2024-04-13T03:49:24.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:24 smithi188 ceph-mon[36398]: pgmap v460: 129 pgs: 129 active+clean; 864 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 606 op/s 2024-04-13T03:49:25.285 INFO:tasks.workunit.client.0.smithi016.stdout: 10 32473 31.56 MB/sec warmup 61 sec latency 208.930 ms 2024-04-13T03:49:26.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 32859 31.39 MB/sec warmup 62 sec latency 519.475 ms 2024-04-13T03:49:27.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 33516 31.43 MB/sec warmup 63 sec latency 90.470 ms 2024-04-13T03:49:27.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:26 smithi119 ceph-mon[35835]: pgmap v461: 129 pgs: 129 active+clean; 864 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 10 MiB/s wr, 537 op/s 2024-04-13T03:49:27.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:26 smithi016 ceph-mon[31772]: pgmap v461: 129 pgs: 129 active+clean; 864 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 10 MiB/s wr, 537 op/s 2024-04-13T03:49:27.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:26 smithi188 ceph-mon[36398]: pgmap v461: 129 pgs: 129 active+clean; 864 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 10 MiB/s wr, 537 op/s 2024-04-13T03:49:28.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 34349 31.64 MB/sec warmup 64 sec latency 95.698 ms 2024-04-13T03:49:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:28 smithi188 ceph-mon[36398]: pgmap v462: 129 pgs: 129 active+clean; 894 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 16 MiB/s wr, 836 op/s 2024-04-13T03:49:29.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 35196 31.80 MB/sec warmup 65 sec latency 67.577 ms 2024-04-13T03:49:29.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:28 smithi119 ceph-mon[35835]: pgmap v462: 129 pgs: 129 active+clean; 894 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 16 MiB/s wr, 836 op/s 2024-04-13T03:49:29.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:28 smithi016 ceph-mon[31772]: pgmap v462: 129 pgs: 129 active+clean; 894 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 16 MiB/s wr, 836 op/s 2024-04-13T03:49:30.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 36040 32.15 MB/sec warmup 66 sec latency 48.543 ms 2024-04-13T03:49:31.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 36426 31.95 MB/sec warmup 67 sec latency 168.358 ms 2024-04-13T03:49:31.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:30 smithi119 ceph-mon[35835]: pgmap v463: 129 pgs: 129 active+clean; 894 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 611 op/s 2024-04-13T03:49:31.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:30 smithi016 ceph-mon[31772]: pgmap v463: 129 pgs: 129 active+clean; 894 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 611 op/s 2024-04-13T03:49:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:30 smithi188 ceph-mon[36398]: pgmap v463: 129 pgs: 129 active+clean; 894 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 611 op/s 2024-04-13T03:49:32.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 36853 31.76 MB/sec warmup 68 sec latency 128.585 ms 2024-04-13T03:49:33.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 37602 31.89 MB/sec warmup 69 sec latency 56.124 ms 2024-04-13T03:49:33.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:32 smithi119 ceph-mon[35835]: pgmap v464: 129 pgs: 129 active+clean; 928 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 16 MiB/s wr, 900 op/s 2024-04-13T03:49:33.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:32 smithi016 ceph-mon[31772]: pgmap v464: 129 pgs: 129 active+clean; 928 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 16 MiB/s wr, 900 op/s 2024-04-13T03:49:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:32 smithi188 ceph-mon[36398]: pgmap v464: 129 pgs: 129 active+clean; 928 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 16 MiB/s wr, 900 op/s 2024-04-13T03:49:34.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 38404 32.00 MB/sec warmup 70 sec latency 62.370 ms 2024-04-13T03:49:35.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 39197 32.23 MB/sec warmup 71 sec latency 60.998 ms 2024-04-13T03:49:35.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:34 smithi119 ceph-mon[35835]: pgmap v465: 129 pgs: 129 active+clean; 928 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 588 op/s 2024-04-13T03:49:35.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:34 smithi016 ceph-mon[31772]: pgmap v465: 129 pgs: 129 active+clean; 928 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 588 op/s 2024-04-13T03:49:35.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:34 smithi188 ceph-mon[36398]: pgmap v465: 129 pgs: 129 active+clean; 928 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 588 op/s 2024-04-13T03:49:36.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 39643 32.01 MB/sec warmup 72 sec latency 357.756 ms 2024-04-13T03:49:37.286 INFO:tasks.workunit.client.0.smithi016.stdout: 10 40253 32.13 MB/sec warmup 73 sec latency 92.735 ms 2024-04-13T03:49:37.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:36 smithi119 ceph-mon[35835]: pgmap v466: 129 pgs: 129 active+clean; 928 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 588 op/s 2024-04-13T03:49:37.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:36 smithi016 ceph-mon[31772]: pgmap v466: 129 pgs: 129 active+clean; 928 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 588 op/s 2024-04-13T03:49:37.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:36 smithi188 ceph-mon[36398]: pgmap v466: 129 pgs: 129 active+clean; 928 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 588 op/s 2024-04-13T03:49:38.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 40842 32.12 MB/sec warmup 74 sec latency 66.476 ms 2024-04-13T03:49:39.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 41491 32.13 MB/sec warmup 75 sec latency 50.280 ms 2024-04-13T03:49:39.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:38 smithi119 ceph-mon[35835]: pgmap v467: 129 pgs: 129 active+clean; 955 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 15 MiB/s wr, 868 op/s 2024-04-13T03:49:39.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:38 smithi016 ceph-mon[31772]: pgmap v467: 129 pgs: 129 active+clean; 955 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 15 MiB/s wr, 868 op/s 2024-04-13T03:49:39.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:38 smithi188 ceph-mon[36398]: pgmap v467: 129 pgs: 129 active+clean; 955 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 15 MiB/s wr, 868 op/s 2024-04-13T03:49:40.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 42126 32.13 MB/sec warmup 76 sec latency 182.081 ms 2024-04-13T03:49:41.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 42600 31.93 MB/sec warmup 77 sec latency 254.116 ms 2024-04-13T03:49:41.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:40 smithi119 ceph-mon[35835]: pgmap v468: 129 pgs: 129 active+clean; 955 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 568 op/s 2024-04-13T03:49:41.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:40 smithi016 ceph-mon[31772]: pgmap v468: 129 pgs: 129 active+clean; 955 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 568 op/s 2024-04-13T03:49:41.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:40 smithi188 ceph-mon[36398]: pgmap v468: 129 pgs: 129 active+clean; 955 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 568 op/s 2024-04-13T03:49:42.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 43054 31.95 MB/sec warmup 78 sec latency 269.616 ms 2024-04-13T03:49:43.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 43842 32.11 MB/sec warmup 79 sec latency 90.512 ms 2024-04-13T03:49:43.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:42 smithi119 ceph-mon[35835]: pgmap v469: 129 pgs: 129 active+clean; 996 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 15 MiB/s wr, 858 op/s 2024-04-13T03:49:43.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:42 smithi016 ceph-mon[31772]: pgmap v469: 129 pgs: 129 active+clean; 996 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 15 MiB/s wr, 858 op/s 2024-04-13T03:49:43.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:42 smithi188 ceph-mon[36398]: pgmap v469: 129 pgs: 129 active+clean; 996 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 15 MiB/s wr, 858 op/s 2024-04-13T03:49:44.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 44526 32.20 MB/sec warmup 80 sec latency 153.608 ms 2024-04-13T03:49:45.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 45158 32.22 MB/sec warmup 81 sec latency 40.649 ms 2024-04-13T03:49:45.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:44 smithi119 ceph-mon[35835]: pgmap v470: 129 pgs: 129 active+clean; 996 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 569 op/s 2024-04-13T03:49:45.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:44 smithi016 ceph-mon[31772]: pgmap v470: 129 pgs: 129 active+clean; 996 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 569 op/s 2024-04-13T03:49:45.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:44 smithi188 ceph-mon[36398]: pgmap v470: 129 pgs: 129 active+clean; 996 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 569 op/s 2024-04-13T03:49:46.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 45685 32.16 MB/sec warmup 82 sec latency 133.746 ms 2024-04-13T03:49:47.287 INFO:tasks.workunit.client.0.smithi016.stdout: 10 45938 31.88 MB/sec warmup 83 sec latency 628.631 ms 2024-04-13T03:49:47.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:46 smithi119 ceph-mon[35835]: pgmap v471: 129 pgs: 129 active+clean; 996 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 569 op/s 2024-04-13T03:49:47.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:46 smithi016 ceph-mon[31772]: pgmap v471: 129 pgs: 129 active+clean; 996 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 569 op/s 2024-04-13T03:49:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:46 smithi188 ceph-mon[36398]: pgmap v471: 129 pgs: 129 active+clean; 996 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 9.8 MiB/s wr, 569 op/s 2024-04-13T03:49:48.289 INFO:tasks.workunit.client.0.smithi016.stdout: 10 46028 31.56 MB/sec warmup 84 sec latency 1101.693 ms 2024-04-13T03:49:49.290 INFO:tasks.workunit.client.0.smithi016.stdout: 10 46363 31.36 MB/sec warmup 85 sec latency 577.636 ms 2024-04-13T03:49:49.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:48 smithi119 ceph-mon[35835]: pgmap v472: 129 pgs: 129 active+clean; 983 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 842 op/s 2024-04-13T03:49:49.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:48 smithi016 ceph-mon[31772]: pgmap v472: 129 pgs: 129 active+clean; 983 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 842 op/s 2024-04-13T03:49:49.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:48 smithi188 ceph-mon[36398]: pgmap v472: 129 pgs: 129 active+clean; 983 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 842 op/s 2024-04-13T03:49:50.289 INFO:tasks.workunit.client.0.smithi016.stdout: 10 46708 31.25 MB/sec warmup 86 sec latency 138.074 ms 2024-04-13T03:49:51.290 INFO:tasks.workunit.client.0.smithi016.stdout: 10 46997 31.08 MB/sec warmup 87 sec latency 258.912 ms 2024-04-13T03:49:51.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:50 smithi119 ceph-mon[35835]: pgmap v473: 129 pgs: 129 active+clean; 983 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 8.0 MiB/s wr, 562 op/s 2024-04-13T03:49:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:50 smithi016 ceph-mon[31772]: pgmap v473: 129 pgs: 129 active+clean; 983 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 8.0 MiB/s wr, 562 op/s 2024-04-13T03:49:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:50 smithi188 ceph-mon[36398]: pgmap v473: 129 pgs: 129 active+clean; 983 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 8.0 MiB/s wr, 562 op/s 2024-04-13T03:49:52.290 INFO:tasks.workunit.client.0.smithi016.stdout: 10 47872 31.22 MB/sec warmup 88 sec latency 93.418 ms 2024-04-13T03:49:53.290 INFO:tasks.workunit.client.0.smithi016.stdout: 10 48578 31.45 MB/sec warmup 89 sec latency 76.278 ms 2024-04-13T03:49:53.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:52 smithi119 ceph-mon[35835]: pgmap v474: 129 pgs: 129 active+clean; 983 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 1.05k op/s 2024-04-13T03:49:53.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:52 smithi016 ceph-mon[31772]: pgmap v474: 129 pgs: 129 active+clean; 983 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 1.05k op/s 2024-04-13T03:49:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:52 smithi188 ceph-mon[36398]: pgmap v474: 129 pgs: 129 active+clean; 983 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 MiB/s wr, 1.05k op/s 2024-04-13T03:49:54.290 INFO:tasks.workunit.client.0.smithi016.stdout: 10 49089 31.38 MB/sec warmup 90 sec latency 75.406 ms 2024-04-13T03:49:55.290 INFO:tasks.workunit.client.0.smithi016.stdout: 10 49798 31.35 MB/sec warmup 91 sec latency 41.941 ms 2024-04-13T03:49:55.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:54 smithi119 ceph-mon[35835]: pgmap v475: 129 pgs: 129 active+clean; 979 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 17 MiB/s wr, 804 op/s 2024-04-13T03:49:55.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:54 smithi016 ceph-mon[31772]: pgmap v475: 129 pgs: 129 active+clean; 979 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 17 MiB/s wr, 804 op/s 2024-04-13T03:49:55.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:54 smithi188 ceph-mon[36398]: pgmap v475: 129 pgs: 129 active+clean; 979 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 17 MiB/s wr, 804 op/s 2024-04-13T03:49:56.290 INFO:tasks.workunit.client.0.smithi016.stdout: 10 50568 31.48 MB/sec warmup 92 sec latency 101.328 ms 2024-04-13T03:49:57.291 INFO:tasks.workunit.client.0.smithi016.stdout: 10 51390 31.61 MB/sec warmup 93 sec latency 90.386 ms 2024-04-13T03:49:57.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:56 smithi119 ceph-mon[35835]: pgmap v476: 129 pgs: 129 active+clean; 979 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 17 MiB/s wr, 804 op/s 2024-04-13T03:49:57.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:56 smithi016 ceph-mon[31772]: pgmap v476: 129 pgs: 129 active+clean; 979 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 17 MiB/s wr, 804 op/s 2024-04-13T03:49:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:56 smithi188 ceph-mon[36398]: pgmap v476: 129 pgs: 129 active+clean; 979 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 17 MiB/s wr, 804 op/s 2024-04-13T03:49:58.291 INFO:tasks.workunit.client.0.smithi016.stdout: 10 52162 31.76 MB/sec warmup 94 sec latency 59.892 ms 2024-04-13T03:49:59.291 INFO:tasks.workunit.client.0.smithi016.stdout: 10 52702 31.74 MB/sec warmup 95 sec latency 99.305 ms 2024-04-13T03:49:59.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:49:58 smithi119 ceph-mon[35835]: pgmap v477: 129 pgs: 129 active+clean; 1007 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 19 MiB/s wr, 1.04k op/s 2024-04-13T03:49:59.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:49:58 smithi016 ceph-mon[31772]: pgmap v477: 129 pgs: 129 active+clean; 1007 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 19 MiB/s wr, 1.04k op/s 2024-04-13T03:49:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:49:58 smithi188 ceph-mon[36398]: pgmap v477: 129 pgs: 129 active+clean; 1007 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 19 MiB/s wr, 1.04k op/s 2024-04-13T03:50:00.292 INFO:tasks.workunit.client.0.smithi016.stdout: 10 53416 31.80 MB/sec warmup 96 sec latency 43.528 ms 2024-04-13T03:50:00.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:00 smithi119 ceph-mon[35835]: pgmap v478: 129 pgs: 129 active+clean; 1007 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 17 MiB/s wr, 769 op/s 2024-04-13T03:50:00.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:00 smithi119 ceph-mon[35835]: overall HEALTH_OK 2024-04-13T03:50:00.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:00 smithi016 ceph-mon[31772]: pgmap v478: 129 pgs: 129 active+clean; 1007 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 17 MiB/s wr, 769 op/s 2024-04-13T03:50:00.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:00 smithi016 ceph-mon[31772]: overall HEALTH_OK 2024-04-13T03:50:00.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:00 smithi188 ceph-mon[36398]: pgmap v478: 129 pgs: 129 active+clean; 1007 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 17 MiB/s wr, 769 op/s 2024-04-13T03:50:00.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:00 smithi188 ceph-mon[36398]: overall HEALTH_OK 2024-04-13T03:50:01.292 INFO:tasks.workunit.client.0.smithi016.stdout: 10 53582 31.57 MB/sec warmup 97 sec latency 781.587 ms 2024-04-13T03:50:02.292 INFO:tasks.workunit.client.0.smithi016.stdout: 10 53702 31.28 MB/sec warmup 98 sec latency 1289.327 ms 2024-04-13T03:50:03.292 INFO:tasks.workunit.client.0.smithi016.stdout: 10 53703 30.97 MB/sec warmup 99 sec latency 1276.363 ms 2024-04-13T03:50:03.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:02 smithi119 ceph-mon[35835]: pgmap v479: 129 pgs: 129 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 19 MiB/s wr, 947 op/s 2024-04-13T03:50:03.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:02 smithi016 ceph-mon[31772]: pgmap v479: 129 pgs: 129 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 19 MiB/s wr, 947 op/s 2024-04-13T03:50:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:02 smithi188 ceph-mon[36398]: pgmap v479: 129 pgs: 129 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 19 MiB/s wr, 947 op/s 2024-04-13T03:50:04.292 INFO:tasks.workunit.client.0.smithi016.stdout: 10 53950 30.76 MB/sec warmup 100 sec latency 1930.775 ms 2024-04-13T03:50:05.292 INFO:tasks.workunit.client.0.smithi016.stdout: 10 54259 30.57 MB/sec warmup 101 sec latency 309.468 ms 2024-04-13T03:50:05.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:04 smithi119 ceph-mon[35835]: pgmap v480: 129 pgs: 129 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.9 MiB/s wr, 469 op/s 2024-04-13T03:50:05.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:04 smithi016 ceph-mon[31772]: pgmap v480: 129 pgs: 129 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.9 MiB/s wr, 469 op/s 2024-04-13T03:50:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:04 smithi188 ceph-mon[36398]: pgmap v480: 129 pgs: 129 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.9 MiB/s wr, 469 op/s 2024-04-13T03:50:06.292 INFO:tasks.workunit.client.0.smithi016.stdout: 10 54291 30.31 MB/sec warmup 102 sec latency 311.178 ms 2024-04-13T03:50:07.292 INFO:tasks.workunit.client.0.smithi016.stdout: 10 54331 30.05 MB/sec warmup 103 sec latency 329.342 ms 2024-04-13T03:50:07.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:06 smithi119 ceph-mon[35835]: pgmap v481: 129 pgs: 129 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.7 MiB/s wr, 422 op/s 2024-04-13T03:50:07.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:06 smithi016 ceph-mon[31772]: pgmap v481: 129 pgs: 129 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.7 MiB/s wr, 422 op/s 2024-04-13T03:50:07.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:06 smithi188 ceph-mon[36398]: pgmap v481: 129 pgs: 129 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.7 MiB/s wr, 422 op/s 2024-04-13T03:50:08.293 INFO:tasks.workunit.client.0.smithi016.stdout: 10 54956 30.16 MB/sec warmup 104 sec latency 259.126 ms 2024-04-13T03:50:08.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:07 smithi119 ceph-mon[35835]: osdmap e114: 12 total, 12 up, 12 in 2024-04-13T03:50:08.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:07 smithi016 ceph-mon[31772]: osdmap e114: 12 total, 12 up, 12 in 2024-04-13T03:50:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:07 smithi188 ceph-mon[36398]: osdmap e114: 12 total, 12 up, 12 in 2024-04-13T03:50:09.292 INFO:tasks.workunit.client.0.smithi016.stdout: 10 55651 30.18 MB/sec warmup 105 sec latency 63.327 ms 2024-04-13T03:50:09.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:08 smithi119 ceph-mon[35835]: pgmap v483: 129 pgs: 129 active+clean; 1.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.1 MiB/s wr, 503 op/s 2024-04-13T03:50:09.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:08 smithi016 ceph-mon[31772]: pgmap v483: 129 pgs: 129 active+clean; 1.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.1 MiB/s wr, 503 op/s 2024-04-13T03:50:09.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:08 smithi188 ceph-mon[36398]: pgmap v483: 129 pgs: 129 active+clean; 1.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.1 MiB/s wr, 503 op/s 2024-04-13T03:50:10.294 INFO:tasks.workunit.client.0.smithi016.stdout: 10 56362 30.27 MB/sec warmup 106 sec latency 58.340 ms 2024-04-13T03:50:11.293 INFO:tasks.workunit.client.0.smithi016.stdout: 10 57097 30.42 MB/sec warmup 107 sec latency 48.516 ms 2024-04-13T03:50:11.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:10 smithi119 ceph-mon[35835]: pgmap v484: 129 pgs: 129 active+clean; 1.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.1 MiB/s wr, 503 op/s 2024-04-13T03:50:11.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:10 smithi016 ceph-mon[31772]: pgmap v484: 129 pgs: 129 active+clean; 1.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.1 MiB/s wr, 503 op/s 2024-04-13T03:50:11.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:10 smithi188 ceph-mon[36398]: pgmap v484: 129 pgs: 129 active+clean; 1.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.1 MiB/s wr, 503 op/s 2024-04-13T03:50:12.293 INFO:tasks.workunit.client.0.smithi016.stdout: 10 57743 30.41 MB/sec warmup 108 sec latency 52.616 ms 2024-04-13T03:50:13.293 INFO:tasks.workunit.client.0.smithi016.stdout: 10 58146 30.34 MB/sec warmup 109 sec latency 454.197 ms 2024-04-13T03:50:13.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:12 smithi119 ceph-mon[35835]: pgmap v485: 129 pgs: 129 active+clean; 1.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.9 MiB/s wr, 590 op/s 2024-04-13T03:50:13.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:12 smithi016 ceph-mon[31772]: pgmap v485: 129 pgs: 129 active+clean; 1.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.9 MiB/s wr, 590 op/s 2024-04-13T03:50:13.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:12 smithi188 ceph-mon[36398]: pgmap v485: 129 pgs: 129 active+clean; 1.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.9 MiB/s wr, 590 op/s 2024-04-13T03:50:14.293 INFO:tasks.workunit.client.0.smithi016.stdout: 10 58608 30.26 MB/sec warmup 110 sec latency 191.227 ms 2024-04-13T03:50:14.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:14 smithi119 ceph-mon[35835]: osdmap e115: 12 total, 12 up, 12 in 2024-04-13T03:50:14.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:14 smithi119 ceph-mon[35835]: pgmap v487: 129 pgs: 129 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.1 MiB/s wr, 742 op/s 2024-04-13T03:50:14.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:14 smithi016 ceph-mon[31772]: osdmap e115: 12 total, 12 up, 12 in 2024-04-13T03:50:14.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:14 smithi016 ceph-mon[31772]: pgmap v487: 129 pgs: 129 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.1 MiB/s wr, 742 op/s 2024-04-13T03:50:14.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:14 smithi188 ceph-mon[36398]: osdmap e115: 12 total, 12 up, 12 in 2024-04-13T03:50:14.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:14 smithi188 ceph-mon[36398]: pgmap v487: 129 pgs: 129 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.1 MiB/s wr, 742 op/s 2024-04-13T03:50:15.293 INFO:tasks.workunit.client.0.smithi016.stdout: 10 59323 30.36 MB/sec warmup 111 sec latency 57.429 ms 2024-04-13T03:50:16.293 INFO:tasks.workunit.client.0.smithi016.stdout: 10 59891 30.36 MB/sec warmup 112 sec latency 166.445 ms 2024-04-13T03:50:17.293 INFO:tasks.workunit.client.0.smithi016.stdout: 10 60557 30.38 MB/sec warmup 113 sec latency 68.908 ms 2024-04-13T03:50:17.306 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:16 smithi016 ceph-mon[31772]: pgmap v488: 129 pgs: 129 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.8 MiB/s wr, 385 op/s 2024-04-13T03:50:17.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:16 smithi119 ceph-mon[35835]: pgmap v488: 129 pgs: 129 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.8 MiB/s wr, 385 op/s 2024-04-13T03:50:17.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:16 smithi188 ceph-mon[36398]: pgmap v488: 129 pgs: 129 active+clean; 1.0 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.8 MiB/s wr, 385 op/s 2024-04-13T03:50:18.294 INFO:tasks.workunit.client.0.smithi016.stdout: 10 61216 30.47 MB/sec warmup 114 sec latency 65.209 ms 2024-04-13T03:50:19.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 61337 30.28 MB/sec warmup 115 sec latency 783.497 ms 2024-04-13T03:50:19.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:18 smithi119 ceph-mon[35835]: pgmap v489: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-04-13T03:50:19.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:18 smithi016 ceph-mon[31772]: pgmap v489: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-04-13T03:50:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:18 smithi188 ceph-mon[36398]: pgmap v489: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-04-13T03:50:20.294 INFO:tasks.workunit.client.0.smithi016.stdout: 10 61696 30.17 MB/sec warmup 116 sec latency 937.374 ms 2024-04-13T03:50:21.294 INFO:tasks.workunit.client.0.smithi016.stdout: 10 62039 30.03 MB/sec warmup 117 sec latency 185.138 ms 2024-04-13T03:50:21.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:20 smithi119 ceph-mon[35835]: pgmap v490: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-04-13T03:50:21.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:20 smithi016 ceph-mon[31772]: pgmap v490: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-04-13T03:50:21.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:20 smithi188 ceph-mon[36398]: pgmap v490: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-04-13T03:50:22.294 INFO:tasks.workunit.client.0.smithi016.stdout: 10 62654 30.08 MB/sec warmup 118 sec latency 67.835 ms 2024-04-13T03:50:23.294 INFO:tasks.workunit.client.0.smithi016.stdout: 10 63289 30.16 MB/sec warmup 119 sec latency 86.981 ms 2024-04-13T03:50:23.352 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:22 smithi119 ceph-mon[35835]: pgmap v491: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 19 MiB/s wr, 821 op/s 2024-04-13T03:50:23.353 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:22 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:50:23.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:22 smithi016 ceph-mon[31772]: pgmap v491: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 19 MiB/s wr, 821 op/s 2024-04-13T03:50:23.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:22 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:50:23.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:22 smithi188 ceph-mon[36398]: pgmap v491: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 19 MiB/s wr, 821 op/s 2024-04-13T03:50:23.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:22 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:50:24.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:24 smithi016 ceph-mon[31772]: pgmap v492: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 19 MiB/s wr, 812 op/s 2024-04-13T03:50:24.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:24 smithi188 ceph-mon[36398]: pgmap v492: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 19 MiB/s wr, 812 op/s 2024-04-13T03:50:24.754 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:24 smithi119 ceph-mon[35835]: pgmap v492: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 19 MiB/s wr, 812 op/s 2024-04-13T03:50:25.294 INFO:tasks.workunit.client.0.smithi016.stdout: 10 64602 42.53 MB/sec execute 1 sec latency 66.855 ms 2024-04-13T03:50:25.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:25 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:50:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:25 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:50:25.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:25 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:50:25.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:25 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:50:25.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:25 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:50:25.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:25 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:50:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:25 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:50:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:25 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:50:25.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:25 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:50:26.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 65202 35.19 MB/sec execute 2 sec latency 170.663 ms 2024-04-13T03:50:26.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:26 smithi119 ceph-mon[35835]: pgmap v493: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 16 MiB/s wr, 719 op/s 2024-04-13T03:50:26.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:26 smithi016 ceph-mon[31772]: pgmap v493: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 16 MiB/s wr, 719 op/s 2024-04-13T03:50:26.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:26 smithi188 ceph-mon[36398]: pgmap v493: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 16 MiB/s wr, 719 op/s 2024-04-13T03:50:27.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 65939 39.93 MB/sec execute 3 sec latency 69.953 ms 2024-04-13T03:50:28.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 66547 38.82 MB/sec execute 4 sec latency 78.775 ms 2024-04-13T03:50:29.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 67123 35.28 MB/sec execute 5 sec latency 74.191 ms 2024-04-13T03:50:29.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:28 smithi119 ceph-mon[35835]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 20 MiB/s wr, 955 op/s 2024-04-13T03:50:29.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:28 smithi188 ceph-mon[36398]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 20 MiB/s wr, 955 op/s 2024-04-13T03:50:29.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:28 smithi016 ceph-mon[31772]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 20 MiB/s wr, 955 op/s 2024-04-13T03:50:30.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 67775 35.59 MB/sec execute 6 sec latency 304.588 ms 2024-04-13T03:50:31.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 68377 35.49 MB/sec execute 7 sec latency 305.716 ms 2024-04-13T03:50:31.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:30 smithi119 ceph-mon[35835]: pgmap v495: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 13 MiB/s wr, 650 op/s 2024-04-13T03:50:31.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:30 smithi016 ceph-mon[31772]: pgmap v495: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 13 MiB/s wr, 650 op/s 2024-04-13T03:50:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:30 smithi188 ceph-mon[36398]: pgmap v495: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 13 MiB/s wr, 650 op/s 2024-04-13T03:50:32.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 69044 34.57 MB/sec execute 8 sec latency 322.106 ms 2024-04-13T03:50:33.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 69763 35.90 MB/sec execute 9 sec latency 58.407 ms 2024-04-13T03:50:33.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:32 smithi119 ceph-mon[35835]: pgmap v496: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 16 MiB/s wr, 855 op/s 2024-04-13T03:50:33.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:32 smithi016 ceph-mon[31772]: pgmap v496: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 16 MiB/s wr, 855 op/s 2024-04-13T03:50:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:32 smithi188 ceph-mon[36398]: pgmap v496: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 16 MiB/s wr, 855 op/s 2024-04-13T03:50:34.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 70435 35.61 MB/sec execute 10 sec latency 217.516 ms 2024-04-13T03:50:35.196 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:34 smithi119 ceph-mon[35835]: pgmap v497: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.0 MiB/s wr, 507 op/s 2024-04-13T03:50:35.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 71121 36.34 MB/sec execute 11 sec latency 71.731 ms 2024-04-13T03:50:35.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:34 smithi016 ceph-mon[31772]: pgmap v497: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.0 MiB/s wr, 507 op/s 2024-04-13T03:50:35.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:34 smithi188 ceph-mon[36398]: pgmap v497: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 7.0 MiB/s wr, 507 op/s 2024-04-13T03:50:36.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 71645 34.98 MB/sec execute 12 sec latency 141.744 ms 2024-04-13T03:50:37.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 72225 34.98 MB/sec execute 13 sec latency 200.655 ms 2024-04-13T03:50:37.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:36 smithi119 ceph-mon[35835]: pgmap v498: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 463 op/s 2024-04-13T03:50:37.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:36 smithi016 ceph-mon[31772]: pgmap v498: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 463 op/s 2024-04-13T03:50:37.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:36 smithi188 ceph-mon[36398]: pgmap v498: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.9 MiB/s wr, 463 op/s 2024-04-13T03:50:38.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 72862 35.22 MB/sec execute 14 sec latency 44.544 ms 2024-04-13T03:50:39.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 73592 34.47 MB/sec execute 15 sec latency 101.664 ms 2024-04-13T03:50:39.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:38 smithi119 ceph-mon[35835]: pgmap v499: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.6 MiB/s wr, 684 op/s 2024-04-13T03:50:39.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:38 smithi016 ceph-mon[31772]: pgmap v499: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.6 MiB/s wr, 684 op/s 2024-04-13T03:50:39.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:38 smithi188 ceph-mon[36398]: pgmap v499: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.6 MiB/s wr, 684 op/s 2024-04-13T03:50:40.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 74374 35.82 MB/sec execute 16 sec latency 235.357 ms 2024-04-13T03:50:40.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:39 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:50:40.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:39 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:50:40.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:39 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:50:41.295 INFO:tasks.workunit.client.0.smithi016.stdout: 10 75137 35.97 MB/sec execute 17 sec latency 53.528 ms 2024-04-13T03:50:41.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:40 smithi119 ceph-mon[35835]: pgmap v500: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 5.5 MiB/s wr, 448 op/s 2024-04-13T03:50:41.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:40 smithi016 ceph-mon[31772]: pgmap v500: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 5.5 MiB/s wr, 448 op/s 2024-04-13T03:50:41.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:40 smithi188 ceph-mon[36398]: pgmap v500: 129 pgs: 129 active+clean; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 5.5 MiB/s wr, 448 op/s 2024-04-13T03:50:42.296 INFO:tasks.workunit.client.0.smithi016.stdout: 10 75602 35.31 MB/sec execute 18 sec latency 158.528 ms 2024-04-13T03:50:43.296 INFO:tasks.workunit.client.0.smithi016.stdout: 10 76309 35.38 MB/sec execute 19 sec latency 61.972 ms 2024-04-13T03:50:43.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:42 smithi119 ceph-mon[35835]: pgmap v501: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.8 MiB/s wr, 651 op/s 2024-04-13T03:50:43.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:42 smithi016 ceph-mon[31772]: pgmap v501: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.8 MiB/s wr, 651 op/s 2024-04-13T03:50:43.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:42 smithi188 ceph-mon[36398]: pgmap v501: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.8 MiB/s wr, 651 op/s 2024-04-13T03:50:44.296 INFO:tasks.workunit.client.0.smithi016.stdout: 10 77104 35.77 MB/sec execute 20 sec latency 49.216 ms 2024-04-13T03:50:45.296 INFO:tasks.workunit.client.0.smithi016.stdout: 10 77767 35.99 MB/sec execute 21 sec latency 110.822 ms 2024-04-13T03:50:45.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:44 smithi119 ceph-mon[35835]: pgmap v502: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.7 MiB/s wr, 474 op/s 2024-04-13T03:50:45.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:44 smithi016 ceph-mon[31772]: pgmap v502: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.7 MiB/s wr, 474 op/s 2024-04-13T03:50:45.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:44 smithi188 ceph-mon[36398]: pgmap v502: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.7 MiB/s wr, 474 op/s 2024-04-13T03:50:46.296 INFO:tasks.workunit.client.0.smithi016.stdout: 10 78327 35.43 MB/sec execute 22 sec latency 107.406 ms 2024-04-13T03:50:47.296 INFO:tasks.workunit.client.0.smithi016.stdout: 10 78995 35.48 MB/sec execute 23 sec latency 146.211 ms 2024-04-13T03:50:47.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:46 smithi119 ceph-mon[35835]: pgmap v503: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 453 op/s 2024-04-13T03:50:47.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:46 smithi016 ceph-mon[31772]: pgmap v503: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 453 op/s 2024-04-13T03:50:47.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:46 smithi188 ceph-mon[36398]: pgmap v503: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.4 MiB/s wr, 453 op/s 2024-04-13T03:50:48.297 INFO:tasks.workunit.client.0.smithi016.stdout: 10 79763 35.92 MB/sec execute 24 sec latency 55.506 ms 2024-04-13T03:50:48.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:48 smithi119 ceph-mon[35835]: pgmap v504: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.8 MiB/s wr, 688 op/s 2024-04-13T03:50:48.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:48 smithi016 ceph-mon[31772]: pgmap v504: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.8 MiB/s wr, 688 op/s 2024-04-13T03:50:48.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:48 smithi188 ceph-mon[36398]: pgmap v504: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.8 MiB/s wr, 688 op/s 2024-04-13T03:50:49.297 INFO:tasks.workunit.client.0.smithi016.stdout: 10 80424 35.99 MB/sec execute 25 sec latency 51.165 ms 2024-04-13T03:50:50.297 INFO:tasks.workunit.client.0.smithi016.stdout: 10 80676 35.14 MB/sec execute 26 sec latency 474.046 ms 2024-04-13T03:50:51.297 INFO:tasks.workunit.client.0.smithi016.stdout: 10 81058 34.40 MB/sec execute 27 sec latency 109.553 ms 2024-04-13T03:50:51.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:50 smithi119 ceph-mon[35835]: pgmap v505: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 467 op/s 2024-04-13T03:50:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:50 smithi016 ceph-mon[31772]: pgmap v505: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 467 op/s 2024-04-13T03:50:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:50 smithi188 ceph-mon[36398]: pgmap v505: 129 pgs: 129 active+clean; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.1 MiB/s wr, 467 op/s 2024-04-13T03:50:52.297 INFO:tasks.workunit.client.0.smithi016.stdout: 10 81502 33.99 MB/sec execute 28 sec latency 157.372 ms 2024-04-13T03:50:53.297 INFO:tasks.workunit.client.0.smithi016.stdout: 10 82395 34.21 MB/sec execute 29 sec latency 63.077 ms 2024-04-13T03:50:53.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:52 smithi119 ceph-mon[35835]: pgmap v506: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 774 op/s 2024-04-13T03:50:53.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:52 smithi016 ceph-mon[31772]: pgmap v506: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 774 op/s 2024-04-13T03:50:53.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:52 smithi188 ceph-mon[36398]: pgmap v506: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 774 op/s 2024-04-13T03:50:54.297 INFO:tasks.workunit.client.0.smithi016.stdout: 10 83138 34.78 MB/sec execute 30 sec latency 88.470 ms 2024-04-13T03:50:55.297 INFO:tasks.workunit.client.0.smithi016.stdout: 10 83794 34.85 MB/sec execute 31 sec latency 58.124 ms 2024-04-13T03:50:55.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:54 smithi119 ceph-mon[35835]: pgmap v507: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 597 op/s 2024-04-13T03:50:55.355 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:54 smithi016 ceph-mon[31772]: pgmap v507: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 597 op/s 2024-04-13T03:50:55.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:54 smithi188 ceph-mon[36398]: pgmap v507: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 597 op/s 2024-04-13T03:50:56.297 INFO:tasks.workunit.client.0.smithi016.stdout: 10 83823 33.77 MB/sec execute 32 sec latency 750.249 ms 2024-04-13T03:50:57.298 INFO:tasks.workunit.client.0.smithi016.stdout: 10 83961 32.97 MB/sec execute 33 sec latency 1393.967 ms 2024-04-13T03:50:57.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:56 smithi119 ceph-mon[35835]: pgmap v508: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 10 MiB/s wr, 567 op/s 2024-04-13T03:50:57.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:56 smithi016 ceph-mon[31772]: pgmap v508: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 10 MiB/s wr, 567 op/s 2024-04-13T03:50:57.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:56 smithi188 ceph-mon[36398]: pgmap v508: 129 pgs: 129 active+clean; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 10 MiB/s wr, 567 op/s 2024-04-13T03:50:58.298 INFO:tasks.workunit.client.0.smithi016.stdout: 10 84217 32.44 MB/sec execute 34 sec latency 169.768 ms 2024-04-13T03:50:59.300 INFO:tasks.workunit.client.0.smithi016.stdout: 10 84503 32.01 MB/sec execute 35 sec latency 92.680 ms 2024-04-13T03:50:59.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:58 smithi119 ceph-mon[35835]: pgmap v509: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 24 MiB/s wr, 938 op/s 2024-04-13T03:50:59.356 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:58 smithi016 ceph-mon[31772]: pgmap v509: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 24 MiB/s wr, 938 op/s 2024-04-13T03:50:59.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:58 smithi188 ceph-mon[36398]: pgmap v509: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 24 MiB/s wr, 938 op/s 2024-04-13T03:51:00.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85009 31.64 MB/sec execute 36 sec latency 135.026 ms 2024-04-13T03:51:00.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:50:59 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:00.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:50:59 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:00.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:50:59 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:01.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85741 31.92 MB/sec execute 37 sec latency 139.241 ms 2024-04-13T03:51:01.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:00 smithi119 ceph-mon[35835]: pgmap v510: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 20 MiB/s wr, 702 op/s 2024-04-13T03:51:01.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:00 smithi016 ceph-mon[31772]: pgmap v510: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 20 MiB/s wr, 702 op/s 2024-04-13T03:51:01.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:00 smithi188 ceph-mon[36398]: pgmap v510: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 20 MiB/s wr, 702 op/s 2024-04-13T03:51:02.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 85920 31.27 MB/sec execute 38 sec latency 532.842 ms 2024-04-13T03:51:03.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 86014 30.57 MB/sec execute 39 sec latency 398.241 ms 2024-04-13T03:51:03.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:02 smithi119 ceph-mon[35835]: pgmap v511: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 22 MiB/s wr, 1.02k op/s 2024-04-13T03:51:03.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:02 smithi119 ceph-mon[35835]: osdmap e116: 12 total, 12 up, 12 in 2024-04-13T03:51:03.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:02 smithi016 ceph-mon[31772]: pgmap v511: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 22 MiB/s wr, 1.02k op/s 2024-04-13T03:51:03.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:02 smithi016 ceph-mon[31772]: osdmap e116: 12 total, 12 up, 12 in 2024-04-13T03:51:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:02 smithi188 ceph-mon[36398]: pgmap v511: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 22 MiB/s wr, 1.02k op/s 2024-04-13T03:51:03.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:02 smithi188 ceph-mon[36398]: osdmap e116: 12 total, 12 up, 12 in 2024-04-13T03:51:04.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 86040 29.89 MB/sec execute 40 sec latency 375.075 ms 2024-04-13T03:51:05.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 86215 29.51 MB/sec execute 41 sec latency 345.874 ms 2024-04-13T03:51:05.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:05 smithi119 ceph-mon[35835]: pgmap v513: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 19 MiB/s wr, 884 op/s 2024-04-13T03:51:05.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:05 smithi016 ceph-mon[31772]: pgmap v513: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 19 MiB/s wr, 884 op/s 2024-04-13T03:51:05.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:05 smithi188 ceph-mon[36398]: pgmap v513: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 19 MiB/s wr, 884 op/s 2024-04-13T03:51:06.298 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:06 smithi119 ceph-mon[35835]: pgmap v514: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 19 MiB/s wr, 884 op/s 2024-04-13T03:51:06.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 86709 29.35 MB/sec execute 42 sec latency 529.772 ms 2024-04-13T03:51:06.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:06 smithi016 ceph-mon[31772]: pgmap v514: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 19 MiB/s wr, 884 op/s 2024-04-13T03:51:06.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:06 smithi188 ceph-mon[36398]: pgmap v514: 129 pgs: 129 active+clean; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 19 MiB/s wr, 884 op/s 2024-04-13T03:51:07.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 87378 29.43 MB/sec execute 43 sec latency 164.998 ms 2024-04-13T03:51:08.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88094 29.86 MB/sec execute 44 sec latency 99.233 ms 2024-04-13T03:51:08.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:08 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:08.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:08 smithi119 ceph-mon[35835]: pgmap v515: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.2 MiB/s wr, 771 op/s 2024-04-13T03:51:08.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:08 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:08.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:08 smithi016 ceph-mon[31772]: pgmap v515: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.2 MiB/s wr, 771 op/s 2024-04-13T03:51:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:08 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:08.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:08 smithi188 ceph-mon[36398]: pgmap v515: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.2 MiB/s wr, 771 op/s 2024-04-13T03:51:09.299 INFO:tasks.workunit.client.0.smithi016.stdout: 10 88692 29.89 MB/sec execute 45 sec latency 150.749 ms 2024-04-13T03:51:09.449 INFO:tasks.workunit.client.0.smithi016.stdout:[90251] write failed on handle 12175 (Resource temporarily unavailable) 2024-04-13T03:51:09.449 INFO:tasks.workunit.client.0.smithi016.stdout:Child failed with status 1 2024-04-13T03:51:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T03:51:09.470 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-13T03:51:09.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-13T03:51:09.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:09 smithi016 ceph-mon[31772]: osdmap e117: 12 total, 12 up, 12 in 2024-04-13T03:51:09.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:09 smithi119 ceph-mon[35835]: osdmap e117: 12 total, 12 up, 12 in 2024-04-13T03:51:09.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:09 smithi188 ceph-mon[36398]: osdmap e117: 12 total, 12 up, 12 in 2024-04-13T03:51:10.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:10 smithi119 ceph-mon[35835]: pgmap v517: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 489 op/s 2024-04-13T03:51:10.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:10 smithi016 ceph-mon[31772]: pgmap v517: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 489 op/s 2024-04-13T03:51:10.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:10 smithi188 ceph-mon[36398]: pgmap v517: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 489 op/s 2024-04-13T03:51:11.143 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_218a691417ef57c5d22a8d5cf3c5f41f7b542838/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_218a691417ef57c5d22a8d5cf3c5f41f7b542838/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_218a691417ef57c5d22a8d5cf3c5f41f7b542838/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-13T03:51:11.386 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=df04e36084ae4a3cb22dd023563280d6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_218a691417ef57c5d22a8d5cf3c5f41f7b542838/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_218a691417ef57c5d22a8d5cf3c5f41f7b542838/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_218a691417ef57c5d22a8d5cf3c5f41f7b542838/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-13T03:51:11.389 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-13T03:51:11.400 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-13T03:51:11.401 INFO:teuthology.task.exec:Executing custom commands... 2024-04-13T03:51:11.401 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T03:51:11.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-13T03:51:11.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-13T03:51:11.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-13T03:51:11.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.835+0000 7ffbeda00640 1 -- 172.21.15.16:0/494286718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe81043f0 msgr2=0x7ffbe81047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:11.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.835+0000 7ffbeda00640 1 --2- 172.21.15.16:0/494286718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe81043f0 0x7ffbe81047d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009860 tx=0x7ffbd802bb70 comp rx=0 tx=0).stop 2024-04-13T03:51:11.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.836+0000 7ffbeda00640 1 -- 172.21.15.16:0/494286718 shutdown_connections 2024-04-13T03:51:11.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.836+0000 7ffbeda00640 1 --2- 172.21.15.16:0/494286718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbe81094c0 0x7ffbe8111090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:11.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.836+0000 7ffbeda00640 1 --2- 172.21.15.16:0/494286718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffbe8104da0 0x7ffbe8108de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:11.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.836+0000 7ffbeda00640 1 --2- 172.21.15.16:0/494286718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe81043f0 0x7ffbe81047d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:11.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.836+0000 7ffbeda00640 1 -- 172.21.15.16:0/494286718 >> 172.21.15.16:0/494286718 conn(0x7ffbe80ffc50 msgr2=0x7ffbe8102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:11.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.836+0000 7ffbeda00640 1 -- 172.21.15.16:0/494286718 shutdown_connections 2024-04-13T03:51:11.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.836+0000 7ffbeda00640 1 -- 172.21.15.16:0/494286718 wait complete. 2024-04-13T03:51:11.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.837+0000 7ffbeda00640 1 Processor -- start 2024-04-13T03:51:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.837+0000 7ffbeda00640 1 -- start start 2024-04-13T03:51:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.837+0000 7ffbeda00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe81043f0 0x7ffbe81a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.837+0000 7ffbeda00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffbe81094c0 0x7ffbe81a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.837+0000 7ffbeda00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbe819a810 0x7ffbe819acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.837+0000 7ffbeda00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe8114550 con 0x7ffbe81043f0 2024-04-13T03:51:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.837+0000 7ffbeda00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe81143d0 con 0x7ffbe81094c0 2024-04-13T03:51:11.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.837+0000 7ffbeda00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe81146d0 con 0x7ffbe819a810 2024-04-13T03:51:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.837+0000 7ffbe6a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe81043f0 0x7ffbe81a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe6000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffbe81094c0 0x7ffbe81a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe6000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffbe81094c0 0x7ffbe81a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:57848/0 (socket says 172.21.15.16:57848) 2024-04-13T03:51:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe6000640 1 -- 172.21.15.16:0/119207269 learned_addr learned my addr 172.21.15.16:0/119207269 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe7400640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbe819a810 0x7ffbe819acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe6000640 1 -- 172.21.15.16:0/119207269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbe819a810 msgr2=0x7ffbe819acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe6000640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbe819a810 0x7ffbe819acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:11.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe6000640 1 -- 172.21.15.16:0/119207269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe81043f0 msgr2=0x7ffbe81a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe6000640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe81043f0 0x7ffbe81a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe6000640 1 -- 172.21.15.16:0/119207269 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe819b3a0 con 0x7ffbe81094c0 2024-04-13T03:51:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe7400640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbe819a810 0x7ffbe819acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:51:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbe6000640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffbe81094c0 0x7ffbe81a0b90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9a0 tx=0x7ffbd400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbdf400640 1 -- 172.21.15.16:0/119207269 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400cd80 con 0x7ffbe81094c0 2024-04-13T03:51:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.838+0000 7ffbdf400640 1 -- 172.21.15.16:0/119207269 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd40035f0 con 0x7ffbe81094c0 2024-04-13T03:51:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.839+0000 7ffbdf400640 1 -- 172.21.15.16:0/119207269 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4010670 con 0x7ffbe81094c0 2024-04-13T03:51:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.839+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe819b690 con 0x7ffbe81094c0 2024-04-13T03:51:11.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.839+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe81a77d0 con 0x7ffbe81094c0 2024-04-13T03:51:11.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.840+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbe81058c0 con 0x7ffbe81094c0 2024-04-13T03:51:11.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.841+0000 7ffbdf400640 1 -- 172.21.15.16:0/119207269 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffbd40037a0 con 0x7ffbe81094c0 2024-04-13T03:51:11.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.843+0000 7ffbdf400640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7ffbcc0a3e40 0x7ffbcc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:11.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.843+0000 7ffbdf400640 1 -- 172.21.15.16:0/119207269 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7ffbd40d0e70 con 0x7ffbe81094c0 2024-04-13T03:51:11.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.843+0000 7ffbe6a00640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7ffbcc0a3e40 0x7ffbcc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:11.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.844+0000 7ffbe6a00640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7ffbcc0a3e40 0x7ffbcc0a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802c180 tx=0x7ffbd80056d0 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:11.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:11.847+0000 7ffbdf400640 1 -- 172.21.15.16:0/119207269 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffbe819b690 con 0x7ffbe81094c0 2024-04-13T03:51:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.080+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 --> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7ffbe8104da0 con 0x7ffbcc0a3e40 2024-04-13T03:51:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.097+0000 7ffbdf400640 1 -- 172.21.15.16:0/119207269 <== mgr.25450 v2:172.21.15.16:6800/1691605870 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7ffbe8104da0 con 0x7ffbcc0a3e40 2024-04-13T03:51:12.097 INFO:teuthology.orchestra.run.smithi016.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-13T00:00:00", "created": "2024-04-13T03:34:46", "first": "2024-04-13T03:35:00", "last": "2024-04-13T03:51:01", "last_pruned": "2024-04-13T03:51:01", "created_count": 17, "pruned_count": 11, "active": true} 2024-04-13T03:51:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7ffbcc0a3e40 msgr2=0x7ffbcc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7ffbcc0a3e40 0x7ffbcc0a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802c180 tx=0x7ffbd80056d0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffbe81094c0 msgr2=0x7ffbe81a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffbe81094c0 0x7ffbe81a0b90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9a0 tx=0x7ffbd400ee60 comp rx=0 tx=0).stop 2024-04-13T03:51:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 shutdown_connections 2024-04-13T03:51:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7ffbcc0a3e40 0x7ffbcc0a6300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffbe819a810 0x7ffbe819acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffbe81094c0 0x7ffbe81a0b90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 --2- 172.21.15.16:0/119207269 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbe81043f0 0x7ffbe81a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 >> 172.21.15.16:0/119207269 conn(0x7ffbe80ffc50 msgr2=0x7ffbe8101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 shutdown_connections 2024-04-13T03:51:12.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.100+0000 7ffbeda00640 1 -- 172.21.15.16:0/119207269 wait complete. 2024-04-13T03:51:12.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-13T03:51:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.341+0000 7fc3eaa00640 1 -- 172.21.15.16:0/4114639646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3e4108e80 msgr2=0x7fc3e41152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.341+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/4114639646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3e4108e80 0x7fc3e41152c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009cd0 tx=0x7fc3dc02c0e0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.341+0000 7fc3eaa00640 1 -- 172.21.15.16:0/4114639646 shutdown_connections 2024-04-13T03:51:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.341+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/4114639646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3e4108e80 0x7fc3e41152c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.341+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/4114639646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3e4108500 0x7fc3e4108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.341+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/4114639646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e410e490 0x7fc3e410e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.341+0000 7fc3eaa00640 1 -- 172.21.15.16:0/4114639646 >> 172.21.15.16:0/4114639646 conn(0x7fc3e4077da0 msgr2=0x7fc3e4106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.341+0000 7fc3eaa00640 1 -- 172.21.15.16:0/4114639646 shutdown_connections 2024-04-13T03:51:12.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.341+0000 7fc3eaa00640 1 -- 172.21.15.16:0/4114639646 wait complete. 2024-04-13T03:51:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.342+0000 7fc3eaa00640 1 Processor -- start 2024-04-13T03:51:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.342+0000 7fc3eaa00640 1 -- start start 2024-04-13T03:51:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3eaa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e4108500 0x7fc3e41a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:12.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3eaa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3e4108e80 0x7fc3e41a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3eaa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3e410e490 0x7fc3e419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3eaa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41187c0 con 0x7fc3e4108500 2024-04-13T03:51:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3eaa00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e4118640 con 0x7fc3e4108e80 2024-04-13T03:51:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3eaa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e4118940 con 0x7fc3e410e490 2024-04-13T03:51:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3e9600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e4108500 0x7fc3e41a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3ea000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3e410e490 0x7fc3e419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3e9600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e4108500 0x7fc3e41a48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52756/0 (socket says 172.21.15.16:52756) 2024-04-13T03:51:12.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3e9600640 1 -- 172.21.15.16:0/210899289 learned_addr learned my addr 172.21.15.16:0/210899289 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3e8c00640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3e4108e80 0x7fc3e41a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3ea000640 1 -- 172.21.15.16:0/210899289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3e4108e80 msgr2=0x7fc3e41a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3ea000640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3e4108e80 0x7fc3e41a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3ea000640 1 -- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e4108500 msgr2=0x7fc3e41a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3ea000640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e4108500 0x7fc3e41a48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3ea000640 1 -- 172.21.15.16:0/210899289 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e419f220 con 0x7fc3e410e490 2024-04-13T03:51:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.343+0000 7fc3e9600640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e4108500 0x7fc3e41a48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:51:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.344+0000 7fc3ea000640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3e410e490 0x7fc3e419eaa0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc037040 tx=0x7fc3dc032460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:12.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.344+0000 7fc3e2000640 1 -- 172.21.15.16:0/210899289 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc039040 con 0x7fc3e410e490 2024-04-13T03:51:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.344+0000 7fc3e2000640 1 -- 172.21.15.16:0/210899289 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3dc044070 con 0x7fc3e410e490 2024-04-13T03:51:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.344+0000 7fc3e2000640 1 -- 172.21.15.16:0/210899289 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0042a0 con 0x7fc3e410e490 2024-04-13T03:51:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.344+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e419f4b0 con 0x7fc3e410e490 2024-04-13T03:51:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.344+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e41ab7b0 con 0x7fc3e410e490 2024-04-13T03:51:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.345+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e40505a0 con 0x7fc3e410e490 2024-04-13T03:51:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.347+0000 7fc3e2000640 1 -- 172.21.15.16:0/210899289 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc3dc032590 con 0x7fc3e410e490 2024-04-13T03:51:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.349+0000 7fc3e2000640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc3cc0a3d90 0x7fc3cc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.349+0000 7fc3e2000640 1 -- 172.21.15.16:0/210899289 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fc3dc0f39b0 con 0x7fc3e410e490 2024-04-13T03:51:12.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.349+0000 7fc3e9600640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc3cc0a3d90 0x7fc3cc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:12.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.350+0000 7fc3e9600640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc3cc0a3d90 0x7fc3cc0a6250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc3e419fbe0 tx=0x7fc3d8005f90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:12.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.351+0000 7fc3e2000640 1 -- 172.21.15.16:0/210899289 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3dc0bfd80 con 0x7fc3e410e490 2024-04-13T03:51:12.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.585+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 --> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fc3e4062400 con 0x7fc3cc0a3d90 2024-04-13T03:51:12.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.599+0000 7fc3e2000640 1 -- 172.21.15.16:0/210899289 <== mgr.25450 v2:172.21.15.16:6800/1691605870 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fc3e4062400 con 0x7fc3cc0a3d90 2024-04-13T03:51:12.600 INFO:teuthology.orchestra.run.smithi016.stdout:/ 1m 6m3h 2024-04-13T03:51:12.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc3cc0a3d90 msgr2=0x7fc3cc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc3cc0a3d90 0x7fc3cc0a6250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc3e419fbe0 tx=0x7fc3d8005f90 comp rx=0 tx=0).stop 2024-04-13T03:51:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3e410e490 msgr2=0x7fc3e419eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3e410e490 0x7fc3e419eaa0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc037040 tx=0x7fc3dc032460 comp rx=0 tx=0).stop 2024-04-13T03:51:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 shutdown_connections 2024-04-13T03:51:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc3cc0a3d90 0x7fc3cc0a6250 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3e410e490 0x7fc3e419eaa0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3e4108e80 0x7fc3e41a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 --2- 172.21.15.16:0/210899289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3e4108500 0x7fc3e41a48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 >> 172.21.15.16:0/210899289 conn(0x7fc3e4077da0 msgr2=0x7fc3e4078420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:12.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 shutdown_connections 2024-04-13T03:51:12.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.602+0000 7fc3eaa00640 1 -- 172.21.15.16:0/210899289 wait complete. 2024-04-13T03:51:12.623 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-13T03:51:12.634 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:51:12.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.857+0000 7f8a07000640 1 -- 172.21.15.16:0/2310957434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00108f90 msgr2=0x7f8a001154a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.857+0000 7f8a07000640 1 --2- 172.21.15.16:0/2310957434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00108f90 0x7f8a001154a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009be0 tx=0x7f89f802bef0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.857+0000 7f8a07000640 1 -- 172.21.15.16:0/2310957434 shutdown_connections 2024-04-13T03:51:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.857+0000 7f8a07000640 1 --2- 172.21.15.16:0/2310957434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a00108f90 0x7f8a001154a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.857+0000 7f8a07000640 1 --2- 172.21.15.16:0/2310957434 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001085d0 0x7f8a00108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.857+0000 7f8a07000640 1 --2- 172.21.15.16:0/2310957434 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a0010e670 0x7f8a0010ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.857+0000 7f8a07000640 1 -- 172.21.15.16:0/2310957434 >> 172.21.15.16:0/2310957434 conn(0x7f8a000757c0 msgr2=0x7f8a00106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:12.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.858+0000 7f8a07000640 1 -- 172.21.15.16:0/2310957434 shutdown_connections 2024-04-13T03:51:12.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.858+0000 7f8a07000640 1 -- 172.21.15.16:0/2310957434 wait complete. 2024-04-13T03:51:12.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f8a07000640 1 Processor -- start 2024-04-13T03:51:12.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f8a07000640 1 -- start start 2024-04-13T03:51:12.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f8a07000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001085d0 0x7f8a0019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:12.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f8a07000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00108f90 0x7f8a0019f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:12.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f8a07000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a0010e670 0x7f8a001a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:12.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f8a07000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a001189b0 con 0x7f8a001085d0 2024-04-13T03:51:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f8a07000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8a00118830 con 0x7f8a0010e670 2024-04-13T03:51:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f8a07000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8a00118b30 con 0x7f8a00108f90 2024-04-13T03:51:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f89ffe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001085d0 0x7f8a0019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f89f7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00108f90 0x7f8a0019f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f89ffe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001085d0 0x7f8a0019ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52778/0 (socket says 172.21.15.16:52778) 2024-04-13T03:51:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f89f7e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00108f90 0x7f8a0019f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:44512/0 (socket says 172.21.15.16:44512) 2024-04-13T03:51:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.859+0000 7f89ffe00640 1 -- 172.21.15.16:0/4263542288 learned_addr learned my addr 172.21.15.16:0/4263542288 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.860+0000 7f8a05000640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a0010e670 0x7f8a001a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.860+0000 7f89ffe00640 1 -- 172.21.15.16:0/4263542288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00108f90 msgr2=0x7f8a0019f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.860+0000 7f89ffe00640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00108f90 0x7f8a0019f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.860+0000 7f89ffe00640 1 -- 172.21.15.16:0/4263542288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a0010e670 msgr2=0x7f8a001a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.860+0000 7f89ffe00640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a0010e670 0x7f8a001a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.860+0000 7f89ffe00640 1 -- 172.21.15.16:0/4263542288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a001a3db0 con 0x7f8a001085d0 2024-04-13T03:51:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.860+0000 7f89ffe00640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001085d0 0x7f8a0019ee10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f89f000e9c0 tx=0x7f89f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.860+0000 7f89fd600640 1 -- 172.21.15.16:0/4263542288 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f000cd80 con 0x7f8a001085d0 2024-04-13T03:51:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.861+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a001a40a0 con 0x7f8a001085d0 2024-04-13T03:51:12.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.861+0000 7f89fd600640 1 -- 172.21.15.16:0/4263542288 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f00035f0 con 0x7f8a001085d0 2024-04-13T03:51:12.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.861+0000 7f89fd600640 1 -- 172.21.15.16:0/4263542288 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0010690 con 0x7f8a001085d0 2024-04-13T03:51:12.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.861+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a001ab930 con 0x7f8a001085d0 2024-04-13T03:51:12.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.862+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a00069230 con 0x7f8a001085d0 2024-04-13T03:51:12.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.864+0000 7f89fd600640 1 -- 172.21.15.16:0/4263542288 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f89f00037a0 con 0x7f8a001085d0 2024-04-13T03:51:12.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.865+0000 7f89fd600640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89e40a3da0 0x7f89e40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.866+0000 7f89fd600640 1 -- 172.21.15.16:0/4263542288 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f89f00d0f50 con 0x7f8a001085d0 2024-04-13T03:51:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.866+0000 7f89f7e00640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89e40a3da0 0x7f89e40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:12.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.866+0000 7f89f7e00640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89e40a3da0 0x7f89e40a6260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a00075e30 tx=0x7f89e8009260 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:12.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:12.871+0000 7f89fd600640 1 -- 172.21.15.16:0/4263542288 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89f009d180 con 0x7f8a001085d0 2024-04-13T03:51:13.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.080+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a001a0110 con 0x7f8a001085d0 2024-04-13T03:51:13.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.084+0000 7f89fd600640 1 -- 172.21.15.16:0/4263542288 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+25440 (secure 0 0 0) 0x7f89f00bd0c0 con 0x7f8a001085d0 2024-04-13T03:51:13.085 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:51:13.086 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:51:08.244014+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2756805961":"2024-04-14T03:34:45.365818+0000","172.21.15.16:0/2348781121":"2024-04-14T03:34:45.365818+0000","172.21.15.16:0/1183476525":"2024-04-14T03:34:45.365818+0000","172.21.15.16:6800/1721562410":"2024-04-14T03:34:45.365818+0000","172.21.15.16:0/2242726589":"2024-04-14T03:34:45.365818+0000","172.21.15.16:6801/1721562410":"2024-04-14T03:34:45.365818+0000","172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]},{"pool":4,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:51:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.087+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89e40a3da0 msgr2=0x7f89e40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.087+0000 7f8a07000640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89e40a3da0 0x7f89e40a6260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a00075e30 tx=0x7f89e8009260 comp rx=0 tx=0).stop 2024-04-13T03:51:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.087+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001085d0 msgr2=0x7f8a0019ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.087+0000 7f8a07000640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001085d0 0x7f8a0019ee10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f89f000e9c0 tx=0x7f89f000ee80 comp rx=0 tx=0).stop 2024-04-13T03:51:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.087+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 shutdown_connections 2024-04-13T03:51:13.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.087+0000 7f8a07000640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f89e40a3da0 0x7f89e40a6260 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.088+0000 7f8a07000640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a0010e670 0x7f8a001a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.088+0000 7f8a07000640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a00108f90 0x7f8a0019f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.088+0000 7f8a07000640 1 --2- 172.21.15.16:0/4263542288 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a001085d0 0x7f8a0019ee10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.088+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 >> 172.21.15.16:0/4263542288 conn(0x7f8a000757c0 msgr2=0x7f8a000766c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.088+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 shutdown_connections 2024-04-13T03:51:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.088+0000 7f8a07000640 1 -- 172.21.15.16:0/4263542288 wait complete. 2024-04-13T03:51:13.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:12 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:13.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:12 smithi016 ceph-mon[31772]: pgmap v518: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.4 MiB/s wr, 547 op/s 2024-04-13T03:51:13.109 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-13T03:31:34.802688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-13T03:31:36.036786+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 116, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-13T03:31:39.824413+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 116, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-13T03:51:13.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:51:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:12 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:13.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:12 smithi188 ceph-mon[36398]: pgmap v518: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.4 MiB/s wr, 547 op/s 2024-04-13T03:51:13.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:12 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:13.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:12 smithi119 ceph-mon[35835]: pgmap v518: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.4 MiB/s wr, 547 op/s 2024-04-13T03:51:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.339+0000 7f22aee00640 1 -- 172.21.15.16:0/1864539140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8100ec0 msgr2=0x7f22a81012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.339+0000 7f22aee00640 1 --2- 172.21.15.16:0/1864539140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8100ec0 0x7f22a81012a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f229c009900 tx=0x7f229c02bd10 comp rx=0 tx=0).stop 2024-04-13T03:51:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.339+0000 7f22aee00640 1 -- 172.21.15.16:0/1864539140 shutdown_connections 2024-04-13T03:51:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.339+0000 7f22aee00640 1 --2- 172.21.15.16:0/1864539140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22a81062c0 0x7f22a8106750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.339+0000 7f22aee00640 1 --2- 172.21.15.16:0/1864539140 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a81017e0 0x7f22a8105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.339+0000 7f22aee00640 1 --2- 172.21.15.16:0/1864539140 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a8100ec0 0x7f22a81012a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.339+0000 7f22aee00640 1 -- 172.21.15.16:0/1864539140 >> 172.21.15.16:0/1864539140 conn(0x7f22a80fc930 msgr2=0x7f22a80fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.339+0000 7f22aee00640 1 -- 172.21.15.16:0/1864539140 shutdown_connections 2024-04-13T03:51:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.339+0000 7f22aee00640 1 -- 172.21.15.16:0/1864539140 wait complete. 2024-04-13T03:51:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.340+0000 7f22aee00640 1 Processor -- start 2024-04-13T03:51:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22aee00640 1 -- start start 2024-04-13T03:51:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22aee00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22a8100ec0 0x7f22a819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22aee00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a81017e0 0x7f22a819f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22aee00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a81062c0 0x7f22a81a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22aee00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81188e0 con 0x7f22a81062c0 2024-04-13T03:51:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22aee00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f22a8118760 con 0x7f22a8100ec0 2024-04-13T03:51:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22aee00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f22a8118a60 con 0x7f22a81017e0 2024-04-13T03:51:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22ada00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22a8100ec0 0x7f22a819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22ada00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22a8100ec0 0x7f22a819ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:57896/0 (socket says 172.21.15.16:57896) 2024-04-13T03:51:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22ae400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a81062c0 0x7f22a81a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.341+0000 7f22ada00640 1 -- 172.21.15.16:0/672788640 learned_addr learned my addr 172.21.15.16:0/672788640 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.342+0000 7f22ad000640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a81017e0 0x7f22a819f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.342+0000 7f22ada00640 1 -- 172.21.15.16:0/672788640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a81017e0 msgr2=0x7f22a819f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.342+0000 7f22ada00640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a81017e0 0x7f22a819f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.342+0000 7f22ada00640 1 -- 172.21.15.16:0/672788640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a81062c0 msgr2=0x7f22a81a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.342+0000 7f22ada00640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a81062c0 0x7f22a81a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.342+0000 7f22ada00640 1 -- 172.21.15.16:0/672788640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a81a3e30 con 0x7f22a8100ec0 2024-04-13T03:51:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.342+0000 7f22ada00640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22a8100ec0 0x7f22a819ee40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f229c006010 tx=0x7f229c02cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.342+0000 7f22a6000640 1 -- 172.21.15.16:0/672788640 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c043020 con 0x7f22a8100ec0 2024-04-13T03:51:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.343+0000 7f22a6000640 1 -- 172.21.15.16:0/672788640 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f229c0054e0 con 0x7f22a8100ec0 2024-04-13T03:51:13.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.343+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a81a40c0 con 0x7f22a8100ec0 2024-04-13T03:51:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.343+0000 7f22a6000640 1 -- 172.21.15.16:0/672788640 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c003af0 con 0x7f22a8100ec0 2024-04-13T03:51:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.343+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a81ab9b0 con 0x7f22a8100ec0 2024-04-13T03:51:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.344+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a8068d00 con 0x7f22a8100ec0 2024-04-13T03:51:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.346+0000 7f22a6000640 1 -- 172.21.15.16:0/672788640 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f229c046020 con 0x7f22a8100ec0 2024-04-13T03:51:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.347+0000 7f22a6000640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f22900a3da0 0x7f22900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.347+0000 7f22a6000640 1 -- 172.21.15.16:0/672788640 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f229c0f2ba0 con 0x7f22a8100ec0 2024-04-13T03:51:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.348+0000 7f22ad000640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f22900a3da0 0x7f22900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.348+0000 7f22ad000640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f22900a3da0 0x7f22900a6260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f22a81052e0 tx=0x7f2298009500 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.351+0000 7f22a6000640 1 -- 172.21.15.16:0/672788640 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f229c04d020 con 0x7f22a8100ec0 2024-04-13T03:51:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.561+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f22a81abb80 con 0x7f22a8100ec0 2024-04-13T03:51:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.562+0000 7f22a6000640 1 -- 172.21.15.16:0/672788640 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) ==== 93+0+10 (secure 0 0 0) 0x7f229c0beee0 con 0x7f22a8100ec0 2024-04-13T03:51:13.562 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f22900a3da0 msgr2=0x7f22900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f22900a3da0 0x7f22900a6260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f22a81052e0 tx=0x7f2298009500 comp rx=0 tx=0).stop 2024-04-13T03:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22a8100ec0 msgr2=0x7f22a819ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22a8100ec0 0x7f22a819ee40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f229c006010 tx=0x7f229c02cc20 comp rx=0 tx=0).stop 2024-04-13T03:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 shutdown_connections 2024-04-13T03:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f22900a3da0 0x7f22900a6260 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22a81062c0 0x7f22a81a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a81017e0 0x7f22a819f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 --2- 172.21.15.16:0/672788640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22a8100ec0 0x7f22a819ee40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.565+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 >> 172.21.15.16:0/672788640 conn(0x7f22a80fc930 msgr2=0x7f22a80fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.566+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 shutdown_connections 2024-04-13T03:51:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.566+0000 7f22aee00640 1 -- 172.21.15.16:0/672788640 wait complete. 2024-04-13T03:51:13.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-13T03:51:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.817+0000 7faff7200640 1 -- 172.21.15.16:0/4170612481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff8108f00 msgr2=0x7faff8115370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.817+0000 7faff7200640 1 --2- 172.21.15.16:0/4170612481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff8108f00 0x7faff8115370 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faff0009900 tx=0x7faff002bc40 comp rx=0 tx=0).stop 2024-04-13T03:51:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.817+0000 7faff7200640 1 -- 172.21.15.16:0/4170612481 shutdown_connections 2024-04-13T03:51:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.817+0000 7faff7200640 1 --2- 172.21.15.16:0/4170612481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff8108f00 0x7faff8115370 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.817+0000 7faff7200640 1 --2- 172.21.15.16:0/4170612481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faff8108540 0x7faff81089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.817+0000 7faff7200640 1 --2- 172.21.15.16:0/4170612481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faff810d4c0 0x7faff810d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.817+0000 7faff7200640 1 -- 172.21.15.16:0/4170612481 >> 172.21.15.16:0/4170612481 conn(0x7faff8077de0 msgr2=0x7faff8106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.817+0000 7faff7200640 1 -- 172.21.15.16:0/4170612481 shutdown_connections 2024-04-13T03:51:13.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.817+0000 7faff7200640 1 -- 172.21.15.16:0/4170612481 wait complete. 2024-04-13T03:51:13.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.818+0000 7faff7200640 1 Processor -- start 2024-04-13T03:51:13.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.818+0000 7faff7200640 1 -- start start 2024-04-13T03:51:13.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faff8108540 0x7faff81a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:13.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff7200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faff8108f00 0x7faff81a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:13.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff810d4c0 0x7faff819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:13.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff7200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faff81188e0 con 0x7faff8108540 2024-04-13T03:51:13.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff7200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7faff8118760 con 0x7faff810d4c0 2024-04-13T03:51:13.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff7200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7faff8118a60 con 0x7faff8108f00 2024-04-13T03:51:13.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff810d4c0 0x7faff819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:13.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff5400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faff8108f00 0x7faff81a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:13.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff5e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faff8108540 0x7faff81a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:13.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff810d4c0 0x7faff819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:57910/0 (socket says 172.21.15.16:57910) 2024-04-13T03:51:13.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff6800640 1 -- 172.21.15.16:0/1477499122 learned_addr learned my addr 172.21.15.16:0/1477499122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:13.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff6800640 1 -- 172.21.15.16:0/1477499122 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faff8108f00 msgr2=0x7faff81a4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff6800640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faff8108f00 0x7faff81a4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff6800640 1 -- 172.21.15.16:0/1477499122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faff8108540 msgr2=0x7faff81a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff6800640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faff8108540 0x7faff81a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.819+0000 7faff6800640 1 -- 172.21.15.16:0/1477499122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff819f240 con 0x7faff810d4c0 2024-04-13T03:51:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.820+0000 7faff6800640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff810d4c0 0x7faff819eac0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faff002c180 tx=0x7faff0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.820+0000 7fafeea00640 1 -- 172.21.15.16:0/1477499122 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff0041070 con 0x7faff810d4c0 2024-04-13T03:51:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.820+0000 7fafeea00640 1 -- 172.21.15.16:0/1477499122 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faff002ccd0 con 0x7faff810d4c0 2024-04-13T03:51:13.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.820+0000 7fafeea00640 1 -- 172.21.15.16:0/1477499122 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff0003c90 con 0x7faff810d4c0 2024-04-13T03:51:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.820+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff819f4d0 con 0x7faff810d4c0 2024-04-13T03:51:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.820+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faff81ab780 con 0x7faff810d4c0 2024-04-13T03:51:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.821+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faff8069150 con 0x7faff810d4c0 2024-04-13T03:51:13.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.823+0000 7fafeea00640 1 -- 172.21.15.16:0/1477499122 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faff0044070 con 0x7faff810d4c0 2024-04-13T03:51:13.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.825+0000 7fafeea00640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fafdc0a3da0 0x7fafdc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:13.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.825+0000 7fafeea00640 1 -- 172.21.15.16:0/1477499122 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7faff00f23b0 con 0x7faff810d4c0 2024-04-13T03:51:13.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.825+0000 7faff5e00640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fafdc0a3da0 0x7fafdc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:13.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.826+0000 7faff5e00640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fafdc0a3da0 0x7fafdc0a6260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faff8079df0 tx=0x7fafe8009500 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:13.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:13.829+0000 7fafeea00640 1 -- 172.21.15.16:0/1477499122 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faff00be0a0 con 0x7faff810d4c0 2024-04-13T03:51:14.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.051+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faff81ab980 con 0x7faff810d4c0 2024-04-13T03:51:14.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.051+0000 7fafeea00640 1 -- 172.21.15.16:0/1477499122 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) ==== 104+0+11 (secure 0 0 0) 0x7faff00de070 con 0x7faff810d4c0 2024-04-13T03:51:14.052 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:51:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fafdc0a3da0 msgr2=0x7fafdc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fafdc0a3da0 0x7fafdc0a6260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faff8079df0 tx=0x7fafe8009500 comp rx=0 tx=0).stop 2024-04-13T03:51:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff810d4c0 msgr2=0x7faff819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff810d4c0 0x7faff819eac0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faff002c180 tx=0x7faff0004270 comp rx=0 tx=0).stop 2024-04-13T03:51:14.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 shutdown_connections 2024-04-13T03:51:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fafdc0a3da0 0x7fafdc0a6260 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faff810d4c0 0x7faff819eac0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faff8108f00 0x7faff81a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 --2- 172.21.15.16:0/1477499122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faff8108540 0x7faff81a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 >> 172.21.15.16:0/1477499122 conn(0x7faff8077de0 msgr2=0x7faff8078260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:14.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 shutdown_connections 2024-04-13T03:51:14.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.054+0000 7faff7200640 1 -- 172.21.15.16:0/1477499122 wait complete. 2024-04-13T03:51:14.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-13T03:51:14.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:13 smithi016 ceph-mon[31772]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:51:14.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:13 smithi016 ceph-mon[31772]: from='client.25558 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:51:14.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:13 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4263542288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:51:14.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:13 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/672788640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:51:14.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:13 smithi188 ceph-mon[36398]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:51:14.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:13 smithi188 ceph-mon[36398]: from='client.25558 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:51:14.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:13 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4263542288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:51:14.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:13 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/672788640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:51:14.200 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:13 smithi119 ceph-mon[35835]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:51:14.200 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:13 smithi119 ceph-mon[35835]: from='client.25558 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-13T03:51:14.200 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:13 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4263542288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:51:14.200 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:13 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/672788640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:51:14.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.299+0000 7fba5c000640 1 -- 172.21.15.16:0/1216784715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba54102c00 msgr2=0x7fba54106c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.299+0000 7fba5c000640 1 --2- 172.21.15.16:0/1216784715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba54102c00 0x7fba54106c40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fba40009900 tx=0x7fba4002bc40 comp rx=0 tx=0).stop 2024-04-13T03:51:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.299+0000 7fba5c000640 1 -- 172.21.15.16:0/1216784715 shutdown_connections 2024-04-13T03:51:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.299+0000 7fba5c000640 1 --2- 172.21.15.16:0/1216784715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba54107320 0x7fba5410eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.299+0000 7fba5c000640 1 --2- 172.21.15.16:0/1216784715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba54102c00 0x7fba54106c40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.299+0000 7fba5c000640 1 --2- 172.21.15.16:0/1216784715 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba54102250 0x7fba54102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.299+0000 7fba5c000640 1 -- 172.21.15.16:0/1216784715 >> 172.21.15.16:0/1216784715 conn(0x7fba540fdad0 msgr2=0x7fba540ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.299+0000 7fba5c000640 1 -- 172.21.15.16:0/1216784715 shutdown_connections 2024-04-13T03:51:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.299+0000 7fba5c000640 1 -- 172.21.15.16:0/1216784715 wait complete. 2024-04-13T03:51:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.300+0000 7fba5c000640 1 Processor -- start 2024-04-13T03:51:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba5c000640 1 -- start start 2024-04-13T03:51:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba5c000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba54102250 0x7fba541a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba5c000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba54107320 0x7fba541a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba5c000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba5419a960 0x7fba5419ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba5c000640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fba54112400 con 0x7fba54102250 2024-04-13T03:51:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba5c000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fba54112280 con 0x7fba54107320 2024-04-13T03:51:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba5c000640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fba54112580 con 0x7fba5419a960 2024-04-13T03:51:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba59600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba54102250 0x7fba541a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba59600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba54102250 0x7fba541a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52834/0 (socket says 172.21.15.16:52834) 2024-04-13T03:51:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba59600640 1 -- 172.21.15.16:0/232692819 learned_addr learned my addr 172.21.15.16:0/232692819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:14.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.301+0000 7fba58c00640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba54107320 0x7fba541a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:14.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba5a000640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba5419a960 0x7fba5419ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:14.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba59600640 1 -- 172.21.15.16:0/232692819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba5419a960 msgr2=0x7fba5419ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba59600640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba5419a960 0x7fba5419ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba59600640 1 -- 172.21.15.16:0/232692819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba54107320 msgr2=0x7fba541a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba59600640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba54107320 0x7fba541a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba59600640 1 -- 172.21.15.16:0/232692819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5419b480 con 0x7fba54102250 2024-04-13T03:51:14.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba59600640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba54102250 0x7fba541a07b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fba4800e9f0 tx=0x7fba4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:14.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba52000640 1 -- 172.21.15.16:0/232692819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4800cd80 con 0x7fba54102250 2024-04-13T03:51:14.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba52000640 1 -- 172.21.15.16:0/232692819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba48010910 con 0x7fba54102250 2024-04-13T03:51:14.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5419b770 con 0x7fba54102250 2024-04-13T03:51:14.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.302+0000 7fba52000640 1 -- 172.21.15.16:0/232692819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48018990 con 0x7fba54102250 2024-04-13T03:51:14.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.303+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba541a7a10 con 0x7fba54102250 2024-04-13T03:51:14.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.304+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba54069230 con 0x7fba54102250 2024-04-13T03:51:14.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.306+0000 7fba52000640 1 -- 172.21.15.16:0/232692819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fba48010ac0 con 0x7fba54102250 2024-04-13T03:51:14.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.307+0000 7fba52000640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fba3c0a3da0 0x7fba3c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:14.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.307+0000 7fba52000640 1 -- 172.21.15.16:0/232692819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fba480d0ef0 con 0x7fba54102250 2024-04-13T03:51:14.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.308+0000 7fba58c00640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fba3c0a3da0 0x7fba3c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:14.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.308+0000 7fba58c00640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fba3c0a3da0 0x7fba3c0a6260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba4002c250 tx=0x7fba4002cf90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:14.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.312+0000 7fba52000640 1 -- 172.21.15.16:0/232692819 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba4809cad0 con 0x7fba54102250 2024-04-13T03:51:14.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.527+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fba54102c00 con 0x7fba54102250 2024-04-13T03:51:14.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.528+0000 7fba52000640 1 -- 172.21.15.16:0/232692819 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) ==== 100+0+11 (secure 0 0 0) 0x7fba480a0dd0 con 0x7fba54102250 2024-04-13T03:51:14.529 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-13T03:51:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fba3c0a3da0 msgr2=0x7fba3c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fba3c0a3da0 0x7fba3c0a6260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba4002c250 tx=0x7fba4002cf90 comp rx=0 tx=0).stop 2024-04-13T03:51:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba54102250 msgr2=0x7fba541a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba54102250 0x7fba541a07b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fba4800e9f0 tx=0x7fba4800eeb0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 shutdown_connections 2024-04-13T03:51:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fba3c0a3da0 0x7fba3c0a6260 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba5419a960 0x7fba5419ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba54107320 0x7fba541a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 --2- 172.21.15.16:0/232692819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba54102250 0x7fba541a07b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 >> 172.21.15.16:0/232692819 conn(0x7fba540fdad0 msgr2=0x7fba540fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 shutdown_connections 2024-04-13T03:51:14.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.531+0000 7fba5c000640 1 -- 172.21.15.16:0/232692819 wait complete. 2024-04-13T03:51:14.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-13T03:51:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.828+0000 7f88cfe00640 1 -- 172.21.15.16:0/28078600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88d01039b0 msgr2=0x7f88d010be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.828+0000 7f88cfe00640 1 --2- 172.21.15.16:0/28078600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88d01039b0 0x7f88d010be90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009860 tx=0x7f88b802bc10 comp rx=0 tx=0).stop 2024-04-13T03:51:14.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.828+0000 7f88cfe00640 1 -- 172.21.15.16:0/28078600 shutdown_connections 2024-04-13T03:51:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.828+0000 7f88cfe00640 1 --2- 172.21.15.16:0/28078600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88d010c4b0 0x7f88d010e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.828+0000 7f88cfe00640 1 --2- 172.21.15.16:0/28078600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88d01039b0 0x7f88d010be90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.828+0000 7f88cfe00640 1 --2- 172.21.15.16:0/28078600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d0103090 0x7f88d0103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.828+0000 7f88cfe00640 1 -- 172.21.15.16:0/28078600 >> 172.21.15.16:0/28078600 conn(0x7f88d00fc8d0 msgr2=0x7f88d00fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.829+0000 7f88cfe00640 1 -- 172.21.15.16:0/28078600 shutdown_connections 2024-04-13T03:51:14.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.829+0000 7f88cfe00640 1 -- 172.21.15.16:0/28078600 wait complete. 2024-04-13T03:51:14.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cfe00640 1 Processor -- start 2024-04-13T03:51:14.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cfe00640 1 -- start start 2024-04-13T03:51:14.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d0103090 0x7f88d01a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:14.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cfe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88d01039b0 0x7f88d01a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cfe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88d010c4b0 0x7f88d019a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cfe00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88d0112290 con 0x7f88d0103090 2024-04-13T03:51:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cfe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f88d0112110 con 0x7f88d01039b0 2024-04-13T03:51:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cfe00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f88d0112410 con 0x7f88d010c4b0 2024-04-13T03:51:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d0103090 0x7f88d01a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88c7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88d01039b0 0x7f88d01a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cea00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d0103090 0x7f88d01a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52852/0 (socket says 172.21.15.16:52852) 2024-04-13T03:51:14.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.830+0000 7f88cea00640 1 -- 172.21.15.16:0/3960346700 learned_addr learned my addr 172.21.15.16:0/3960346700 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:14.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.831+0000 7f88cf400640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88d010c4b0 0x7f88d019a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:14.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.831+0000 7f88c7e00640 1 -- 172.21.15.16:0/3960346700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88d010c4b0 msgr2=0x7f88d019a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.831+0000 7f88c7e00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88d010c4b0 0x7f88d019a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.831+0000 7f88c7e00640 1 -- 172.21.15.16:0/3960346700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d0103090 msgr2=0x7f88d01a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:14.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.831+0000 7f88c7e00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d0103090 0x7f88d01a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:14.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.831+0000 7f88c7e00640 1 -- 172.21.15.16:0/3960346700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d019b030 con 0x7f88d01039b0 2024-04-13T03:51:14.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.831+0000 7f88cf400640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88d010c4b0 0x7f88d019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:51:14.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.831+0000 7f88c7e00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88d01039b0 0x7f88d01a0be0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f88b802c150 tx=0x7f88b80344c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:14.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.831+0000 7f88c7400640 1 -- 172.21.15.16:0/3960346700 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b8041020 con 0x7f88d01039b0 2024-04-13T03:51:14.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.832+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d019b2c0 con 0x7f88d01039b0 2024-04-13T03:51:14.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.832+0000 7f88c7400640 1 -- 172.21.15.16:0/3960346700 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88b802ca90 con 0x7f88d01039b0 2024-04-13T03:51:14.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.832+0000 7f88c7400640 1 -- 172.21.15.16:0/3960346700 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b8039040 con 0x7f88d01039b0 2024-04-13T03:51:14.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.832+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d01a74f0 con 0x7f88d01039b0 2024-04-13T03:51:14.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.833+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d006aea0 con 0x7f88d01039b0 2024-04-13T03:51:14.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.835+0000 7f88c7400640 1 -- 172.21.15.16:0/3960346700 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f88b8044020 con 0x7f88d01039b0 2024-04-13T03:51:14.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.836+0000 7f88c7400640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f88b40a3d20 0x7f88b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:14.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.836+0000 7f88c7400640 1 -- 172.21.15.16:0/3960346700 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f88b80f25d0 con 0x7f88d01039b0 2024-04-13T03:51:14.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.837+0000 7f88cea00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f88b40a3d20 0x7f88b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:14.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.837+0000 7f88cea00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f88b40a3d20 0x7f88b40a61e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f88c00058c0 tx=0x7f88c0009260 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:14.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:14.838+0000 7f88c7400640 1 -- 172.21.15.16:0/3960346700 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88b804b020 con 0x7f88d01039b0 2024-04-13T03:51:15.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.051+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f88d0069150 con 0x7f88d01039b0 2024-04-13T03:51:15.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.052+0000 7f88c7400640 1 -- 172.21.15.16:0/3960346700 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v117) ==== 103+0+11 (secure 0 0 0) 0x7f88b80be910 con 0x7f88d01039b0 2024-04-13T03:51:15.053 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:51:15.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.055+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f88b40a3d20 msgr2=0x7f88b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:15.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.055+0000 7f88cfe00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f88b40a3d20 0x7f88b40a61e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f88c00058c0 tx=0x7f88c0009260 comp rx=0 tx=0).stop 2024-04-13T03:51:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.055+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88d01039b0 msgr2=0x7f88d01a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.055+0000 7f88cfe00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88d01039b0 0x7f88d01a0be0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f88b802c150 tx=0x7f88b80344c0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.056+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 shutdown_connections 2024-04-13T03:51:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.056+0000 7f88cfe00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f88b40a3d20 0x7f88b40a61e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.056+0000 7f88cfe00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f88d010c4b0 0x7f88d019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.056+0000 7f88cfe00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88d01039b0 0x7f88d01a0be0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.056+0000 7f88cfe00640 1 --2- 172.21.15.16:0/3960346700 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88d0103090 0x7f88d01a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.056+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 >> 172.21.15.16:0/3960346700 conn(0x7f88d00fc8d0 msgr2=0x7f88d0100cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:15.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.056+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 shutdown_connections 2024-04-13T03:51:15.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.056+0000 7f88cfe00640 1 -- 172.21.15.16:0/3960346700 wait complete. 2024-04-13T03:51:15.070 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:14 smithi119 ceph-mon[35835]: pgmap v519: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.0 MiB/s wr, 490 op/s 2024-04-13T03:51:15.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:14 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1477499122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:51:15.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:14 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/232692819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:51:15.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-13T03:51:15.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:14 smithi016 ceph-mon[31772]: pgmap v519: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.0 MiB/s wr, 490 op/s 2024-04-13T03:51:15.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:14 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1477499122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:51:15.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:14 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/232692819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:51:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:14 smithi188 ceph-mon[36398]: pgmap v519: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.0 MiB/s wr, 490 op/s 2024-04-13T03:51:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:14 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1477499122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:51:15.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:14 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/232692819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:51:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.305+0000 7fb7b2400640 1 -- 172.21.15.16:0/331759885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac112e30 msgr2=0x7fb7ac115220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.305+0000 7fb7b2400640 1 --2- 172.21.15.16:0/331759885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac112e30 0x7fb7ac115220 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009ca0 tx=0x7fb7a402c0f0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.305+0000 7fb7b2400640 1 -- 172.21.15.16:0/331759885 shutdown_connections 2024-04-13T03:51:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.305+0000 7fb7b2400640 1 --2- 172.21.15.16:0/331759885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac112e30 0x7fb7ac115220 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.305+0000 7fb7b2400640 1 --2- 172.21.15.16:0/331759885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac076e10 0x7fb7ac1128f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.305+0000 7fb7b2400640 1 --2- 172.21.15.16:0/331759885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ac0764f0 0x7fb7ac0768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.305+0000 7fb7b2400640 1 -- 172.21.15.16:0/331759885 >> 172.21.15.16:0/331759885 conn(0x7fb7ac0ffcc0 msgr2=0x7fb7ac1020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.305+0000 7fb7b2400640 1 -- 172.21.15.16:0/331759885 shutdown_connections 2024-04-13T03:51:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.305+0000 7fb7b2400640 1 -- 172.21.15.16:0/331759885 wait complete. 2024-04-13T03:51:15.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.307+0000 7fb7b2400640 1 Processor -- start 2024-04-13T03:51:15.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.307+0000 7fb7b2400640 1 -- start start 2024-04-13T03:51:15.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.307+0000 7fb7b2400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac0764f0 0x7fb7ac19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:15.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.307+0000 7fb7b2400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ac076e10 0x7fb7ac19f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:15.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.307+0000 7fb7b2400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac112e30 0x7fb7ac1a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:15.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.307+0000 7fb7b2400640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ac118990 con 0x7fb7ac112e30 2024-04-13T03:51:15.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.307+0000 7fb7b2400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ac118810 con 0x7fb7ac076e10 2024-04-13T03:51:15.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.307+0000 7fb7b2400640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ac118b10 con 0x7fb7ac0764f0 2024-04-13T03:51:15.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7ab400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac0764f0 0x7fb7ac19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:15.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7abe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac112e30 0x7fb7ac1a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:15.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ac076e10 0x7fb7ac19f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:15.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7ab400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac0764f0 0x7fb7ac19ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:44588/0 (socket says 172.21.15.16:44588) 2024-04-13T03:51:15.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7ab400640 1 -- 172.21.15.16:0/2235974691 learned_addr learned my addr 172.21.15.16:0/2235974691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:15.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7ab400640 1 -- 172.21.15.16:0/2235974691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ac076e10 msgr2=0x7fb7ac19f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:15.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7ab400640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ac076e10 0x7fb7ac19f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7ab400640 1 -- 172.21.15.16:0/2235974691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac112e30 msgr2=0x7fb7ac1a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:15.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7ab400640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac112e30 0x7fb7ac1a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:15.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.308+0000 7fb7ab400640 1 -- 172.21.15.16:0/2235974691 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ac1a3eb0 con 0x7fb7ac0764f0 2024-04-13T03:51:15.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.309+0000 7fb7ab400640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac0764f0 0x7fb7ac19ef10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00e9f0 tx=0x7fb79c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:15.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.309+0000 7fb7a8c00640 1 -- 172.21.15.16:0/2235974691 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c00cd80 con 0x7fb7ac0764f0 2024-04-13T03:51:15.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.309+0000 7fb7a8c00640 1 -- 172.21.15.16:0/2235974691 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb79c010910 con 0x7fb7ac0764f0 2024-04-13T03:51:15.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.309+0000 7fb7a8c00640 1 -- 172.21.15.16:0/2235974691 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c018990 con 0x7fb7ac0764f0 2024-04-13T03:51:15.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.309+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ac1a41a0 con 0x7fb7ac0764f0 2024-04-13T03:51:15.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.309+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ac1aba30 con 0x7fb7ac0764f0 2024-04-13T03:51:15.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.310+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7ac069230 con 0x7fb7ac0764f0 2024-04-13T03:51:15.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.313+0000 7fb7a8c00640 1 -- 172.21.15.16:0/2235974691 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb79c010ac0 con 0x7fb7ac0764f0 2024-04-13T03:51:15.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.315+0000 7fb7a8c00640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb7900a3da0 0x7fb7900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:15.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.315+0000 7fb7a8c00640 1 -- 172.21.15.16:0/2235974691 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fb79c0d0ef0 con 0x7fb7ac0764f0 2024-04-13T03:51:15.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.315+0000 7fb7a3e00640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb7900a3da0 0x7fb7900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:15.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.316+0000 7fb7a3e00640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb7900a3da0 0x7fb7900a6260 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac102240 tx=0x7fb794005f90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:15.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.318+0000 7fb7a8c00640 1 -- 172.21.15.16:0/2235974691 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb79c09cad0 con 0x7fb7ac0764f0 2024-04-13T03:51:15.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.585+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fb7ac1a0210 con 0x7fb7ac0764f0 2024-04-13T03:51:15.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.598+0000 7fb7a8c00640 1 -- 172.21.15.16:0/2235974691 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746063 (secure 0 0 0) 0x7fb79c0a0dd0 con 0x7fb7ac0764f0 2024-04-13T03:51:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:51:15.841 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:51:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 26, 2024-04-13T03:51:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "active_gid": 25450, 2024-04-13T03:51:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "active_name": "x", 2024-04-13T03:51:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addrs": { 2024-04-13T03:51:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-13T03:51:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-13T03:51:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2024-04-13T03:51:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1691605870 2024-04-13T03:51:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-13T03:51:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2024-04-13T03:51:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1691605870 2024-04-13T03:51:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "active_addr": "172.21.15.16:6801/1691605870", 2024-04-13T03:51:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "active_change": "2024-04-13T03:34:45.366550+0000", 2024-04-13T03:51:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "active_mgr_features": 4540701547738038271, 2024-04-13T03:51:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "available": true, 2024-04-13T03:51:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "standbys": [ 2024-04-13T03:51:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 25463, 2024-04-13T03:51:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "y", 2024-04-13T03:51:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-04-13T03:51:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-04-13T03:51:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-04-13T03:51:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-04-13T03:51:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:15.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:15.847 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.850 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:15.851 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-04-13T03:51:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-04-13T03:51:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-04-13T03:51:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-04-13T03:51:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-04-13T03:51:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-04-13T03:51:15.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-04-13T03:51:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-04-13T03:51:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-04-13T03:51:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.858 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-04-13T03:51:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-04-13T03:51:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-04-13T03:51:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-04-13T03:51:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-04-13T03:51:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-04-13T03:51:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-04-13T03:51:15.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-04-13T03:51:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-04-13T03:51:15.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-04-13T03:51:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-04-13T03:51:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-04-13T03:51:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-04-13T03:51:15.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-13T03:51:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.865 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-04-13T03:51:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-04-13T03:51:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.866 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-04-13T03:51:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:15.867 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-04-13T03:51:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-04-13T03:51:15.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-04-13T03:51:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-04-13T03:51:15.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-04-13T03:51:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-04-13T03:51:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-04-13T03:51:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-04-13T03:51:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-13T03:51:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-04-13T03:51:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-04-13T03:51:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-13T03:51:15.872 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-13T03:51:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-13T03:51:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-13T03:51:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-04-13T03:51:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-04-13T03:51:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-04-13T03:51:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-04-13T03:51:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-13T03:51:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-04-13T03:51:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-04-13T03:51:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-04-13T03:51:15.875 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-13T03:51:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-13T03:51:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-04-13T03:51:15.876 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-04-13T03:51:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-04-13T03:51:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-04-13T03:51:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-04-13T03:51:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-04-13T03:51:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-13T03:51:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-04-13T03:51:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-04-13T03:51:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-04-13T03:51:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-04-13T03:51:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-13T03:51:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-04-13T03:51:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-04-13T03:51:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-13T03:51:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-13T03:51:15.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-13T03:51:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-13T03:51:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:15.882 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.883 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.884 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:15.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:15.886 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.887 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.888 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-04-13T03:51:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-04-13T03:51:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.890 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-13T03:51:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-13T03:51:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-13T03:51:15.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-04-13T03:51:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-04-13T03:51:15.892 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-04-13T03:51:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-04-13T03:51:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-04-13T03:51:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-04-13T03:51:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-04-13T03:51:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-04-13T03:51:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-04-13T03:51:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.894 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-13T03:51:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:15.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-13T03:51:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-04-13T03:51:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-04-13T03:51:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:15.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-13T03:51:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-13T03:51:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-04-13T03:51:15.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-04-13T03:51:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-13T03:51:15.899 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-04-13T03:51:15.900 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-04-13T03:51:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-04-13T03:51:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-13T03:51:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-13T03:51:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.902 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-04-13T03:51:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-04-13T03:51:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-04-13T03:51:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-13T03:51:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-04-13T03:51:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-04-13T03:51:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-04-13T03:51:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-13T03:51:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-04-13T03:51:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-04-13T03:51:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-13T03:51:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-13T03:51:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-04-13T03:51:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-04-13T03:51:15.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-13T03:51:15.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-04-13T03:51:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-04-13T03:51:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-04-13T03:51:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-13T03:51:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-04-13T03:51:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-04-13T03:51:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-04-13T03:51:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-04-13T03:51:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-04-13T03:51:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-13T03:51:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-04-13T03:51:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-04-13T03:51:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-13T03:51:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-04-13T03:51:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-04-13T03:51:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-13T03:51:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.915 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-04-13T03:51:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-04-13T03:51:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.916 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-13T03:51:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-04-13T03:51:15.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-04-13T03:51:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-13T03:51:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-04-13T03:51:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-04-13T03:51:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-13T03:51:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-04-13T03:51:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-04-13T03:51:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-04-13T03:51:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-13T03:51:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.921 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-04-13T03:51:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-04-13T03:51:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-04-13T03:51:15.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-13T03:51:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-04-13T03:51:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-04-13T03:51:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-13T03:51:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-04-13T03:51:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.925 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-04-13T03:51:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-04-13T03:51:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-13T03:51:15.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-04-13T03:51:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-04-13T03:51:15.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-04-13T03:51:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-13T03:51:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-04-13T03:51:15.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-04-13T03:51:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-04-13T03:51:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-13T03:51:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-04-13T03:51:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-04-13T03:51:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-04-13T03:51:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-13T03:51:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-04-13T03:51:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-04-13T03:51:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-04-13T03:51:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-04-13T03:51:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-13T03:51:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-04-13T03:51:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-04-13T03:51:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-13T03:51:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.936 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-04-13T03:51:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-04-13T03:51:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-04-13T03:51:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.937 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-13T03:51:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.938 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-04-13T03:51:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-04-13T03:51:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-13T03:51:15.939 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-04-13T03:51:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-04-13T03:51:15.940 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-04-13T03:51:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-13T03:51:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-04-13T03:51:15.941 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-04-13T03:51:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-04-13T03:51:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-13T03:51:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-04-13T03:51:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-04-13T03:51:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-04-13T03:51:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.944 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-04-13T03:51:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-04-13T03:51:15.945 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-04-13T03:51:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-13T03:51:15.946 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-04-13T03:51:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-04-13T03:51:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.947 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-04-13T03:51:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-13T03:51:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.948 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-04-13T03:51:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-04-13T03:51:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-04-13T03:51:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.949 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-13T03:51:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-04-13T03:51:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-04-13T03:51:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-04-13T03:51:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-04-13T03:51:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-04-13T03:51:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-04-13T03:51:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.952 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-13T03:51:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-04-13T03:51:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-04-13T03:51:15.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-04-13T03:51:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.955 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-13T03:51:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-04-13T03:51:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-04-13T03:51:15.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-04-13T03:51:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-04-13T03:51:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-04-13T03:51:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.958 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-13T03:51:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-04-13T03:51:15.959 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-04-13T03:51:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-13T03:51:15.960 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:15.962 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-04-13T03:51:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-04-13T03:51:15.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-13T03:51:15.964 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-13T03:51:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.967 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:15.969 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-13T03:51:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-13T03:51:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.971 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-13T03:51:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-13T03:51:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-13T03:51:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-04-13T03:51:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-13T03:51:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-04-13T03:51:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-04-13T03:51:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-04-13T03:51:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-04-13T03:51:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-04-13T03:51:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-13T03:51:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.976 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-04-13T03:51:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-04-13T03:51:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-04-13T03:51:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-13T03:51:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-13T03:51:15.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-04-13T03:51:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-04-13T03:51:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-04-13T03:51:15.979 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-04-13T03:51:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-04-13T03:51:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-04-13T03:51:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-04-13T03:51:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-13T03:51:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.981 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-04-13T03:51:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-04-13T03:51:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-13T03:51:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-13T03:51:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-04-13T03:51:15.983 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-04-13T03:51:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-13T03:51:15.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-04-13T03:51:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-04-13T03:51:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.985 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-13T03:51:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-04-13T03:51:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-04-13T03:51:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.987 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-13T03:51:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-04-13T03:51:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-04-13T03:51:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.988 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-13T03:51:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.989 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-04-13T03:51:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-04-13T03:51:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-13T03:51:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-04-13T03:51:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-04-13T03:51:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-04-13T03:51:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-04-13T03:51:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-04-13T03:51:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-04-13T03:51:15.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-13T03:51:15.994 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-04-13T03:51:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-04-13T03:51:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.995 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-13T03:51:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.996 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-04-13T03:51:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-04-13T03:51:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.997 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-13T03:51:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-04-13T03:51:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-04-13T03:51:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-04-13T03:51:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-04-13T03:51:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-04-13T03:51:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-13T03:51:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-04-13T03:51:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-04-13T03:51:16.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-13T03:51:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-04-13T03:51:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.007 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.009 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.011 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.013 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-04-13T03:51:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-04-13T03:51:16.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-04-13T03:51:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-13T03:51:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-04-13T03:51:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-04-13T03:51:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-04-13T03:51:16.016 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-13T03:51:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-04-13T03:51:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-13T03:51:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-13T03:51:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.018 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.019 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-04-13T03:51:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-13T03:51:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.020 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-13T03:51:16.021 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-13T03:51:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.022 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-04-13T03:51:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-04-13T03:51:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-13T03:51:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-13T03:51:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.025 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-04-13T03:51:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-13T03:51:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.026 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-13T03:51:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-13T03:51:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.028 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.029 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-13T03:51:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-13T03:51:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.030 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-13T03:51:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-13T03:51:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-13T03:51:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-13T03:51:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.033 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-13T03:51:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-13T03:51:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.034 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.035 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-13T03:51:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-13T03:51:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.036 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-13T03:51:16.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-13T03:51:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-13T03:51:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-13T03:51:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.039 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-04-13T03:51:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-13T03:51:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-13T03:51:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.041 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-13T03:51:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-13T03:51:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.042 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-04-13T03:51:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-04-13T03:51:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.044 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-04-13T03:51:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-13T03:51:16.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-13T03:51:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.046 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-13T03:51:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-13T03:51:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.047 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.048 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-13T03:51:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-13T03:51:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.049 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-13T03:51:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-13T03:51:16.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.051 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-13T03:51:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-13T03:51:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.052 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-13T03:51:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-13T03:51:16.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "[]", 2024-04-13T03:51:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "MULTICLUSTER_CONFIG": { 2024-04-13T03:51:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-13T03:51:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "{}", 2024-04-13T03:51:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.056 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.057 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-13T03:51:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-13T03:51:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-13T03:51:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-13T03:51:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.060 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-13T03:51:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-13T03:51:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-13T03:51:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-13T03:51:16.064 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-13T03:51:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-13T03:51:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.066 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-13T03:51:16.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-13T03:51:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-13T03:51:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.070 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-04-13T03:51:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-13T03:51:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.071 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-13T03:51:16.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-13T03:51:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-13T03:51:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.073 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-13T03:51:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-13T03:51:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-13T03:51:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-13T03:51:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-04-13T03:51:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-13T03:51:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-13T03:51:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.077 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-04-13T03:51:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.078 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-04-13T03:51:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-13T03:51:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.079 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-13T03:51:16.080 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-13T03:51:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-13T03:51:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-04-13T03:51:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-13T03:51:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-04-13T03:51:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-13T03:51:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-04-13T03:51:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-13T03:51:16.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-13T03:51:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-13T03:51:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:16.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.088 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.088 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.088 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.088 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-13T03:51:16.088 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-13T03:51:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.089 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.090 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-13T03:51:16.090 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-13T03:51:16.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.090 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:16.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.091 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.091 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.091 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.091 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.091 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-04-13T03:51:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-04-13T03:51:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.093 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.093 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-04-13T03:51:16.093 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-04-13T03:51:16.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.094 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.094 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.094 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.094 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.095 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.095 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-04-13T03:51:16.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-04-13T03:51:16.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.095 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.095 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.095 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-04-13T03:51:16.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.096 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.096 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.097 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-04-13T03:51:16.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-04-13T03:51:16.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.097 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-04-13T03:51:16.097 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.097 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.097 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-13T03:51:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-13T03:51:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.098 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.099 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.099 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.099 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.100 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.100 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.100 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.100 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.101 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.101 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.101 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.101 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.101 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.101 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.101 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.101 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.102 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.102 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.102 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.102 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.102 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.102 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.102 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.103 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.103 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.103 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.103 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.104 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.104 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.104 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.105 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.106 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.108 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-04-13T03:51:16.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-04-13T03:51:16.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.108 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.108 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.108 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.108 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.109 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-04-13T03:51:16.109 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.109 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.109 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-04-13T03:51:16.109 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-04-13T03:51:16.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.109 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.110 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.110 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.110 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.110 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.110 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.110 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.110 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.110 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.111 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-13T03:51:16.111 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-13T03:51:16.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-13T03:51:16.111 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.111 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.112 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.112 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.112 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.112 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-13T03:51:16.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-13T03:51:16.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.113 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.113 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-04-13T03:51:16.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-13T03:51:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-13T03:51:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.115 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.115 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.115 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.115 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.116 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-04-13T03:51:16.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-04-13T03:51:16.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-04-13T03:51:16.116 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.116 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-13T03:51:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-13T03:51:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-04-13T03:51:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-04-13T03:51:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.119 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.119 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.119 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-13T03:51:16.119 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-13T03:51:16.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.119 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.120 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-13T03:51:16.120 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-13T03:51:16.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-13T03:51:16.120 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.120 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.120 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.120 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.121 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-04-13T03:51:16.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-04-13T03:51:16.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.121 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.121 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.121 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.122 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.122 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.122 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.122 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.122 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.122 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-13T03:51:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-04-13T03:51:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-04-13T03:51:16.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-04-13T03:51:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.126 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.126 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.126 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.126 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.126 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.126 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.126 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.126 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.126 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.127 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.127 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.127 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.127 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.127 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.127 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.128 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.128 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.128 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.128 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.128 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.128 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.129 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.129 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.129 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.129 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.130 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.130 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.130 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.130 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.130 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.130 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.130 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.130 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.131 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.131 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.131 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.131 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.132 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.132 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.132 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.132 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.132 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.132 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.132 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.132 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.133 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-04-13T03:51:16.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-04-13T03:51:16.133 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.133 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.133 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.133 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-04-13T03:51:16.133 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-13T03:51:16.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.134 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.134 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-04-13T03:51:16.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-04-13T03:51:16.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.135 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.135 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-04-13T03:51:16.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.135 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-13T03:51:16.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.136 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.136 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-04-13T03:51:16.136 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-04-13T03:51:16.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.136 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-04-13T03:51:16.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.137 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.137 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.137 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-04-13T03:51:16.137 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.137 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.138 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-04-13T03:51:16.138 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-04-13T03:51:16.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.138 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.138 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.138 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-13T03:51:16.138 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.139 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-13T03:51:16.139 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.139 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.139 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-04-13T03:51:16.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-04-13T03:51:16.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.140 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.140 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.140 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-13T03:51:16.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.141 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.141 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:16.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:16.141 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.141 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:16.142 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.142 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.142 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-04-13T03:51:16.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-04-13T03:51:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-04-13T03:51:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-04-13T03:51:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.144 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-13T03:51:16.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.144 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.144 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-04-13T03:51:16.145 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.145 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.145 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.145 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.146 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.146 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.146 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.146 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.146 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.146 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.146 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.146 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.147 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.147 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.147 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.147 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.147 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.147 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.147 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.147 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.148 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.148 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.148 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.148 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.148 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.148 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.148 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.148 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.149 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.149 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.149 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.150 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.150 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.150 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.150 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.150 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.150 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.150 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.150 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.150 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.152 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.152 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.152 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.152 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-13T03:51:16.154 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.154 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.154 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.154 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.154 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.154 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.154 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.155 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-04-13T03:51:16.155 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-04-13T03:51:16.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.155 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.155 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-04-13T03:51:16.155 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.156 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.156 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.156 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.156 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:16.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:16.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.157 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:16.157 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.157 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.157 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.157 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.158 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.158 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-04-13T03:51:16.158 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-04-13T03:51:16.158 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-04-13T03:51:16.158 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.159 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-04-13T03:51:16.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-04-13T03:51:16.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-04-13T03:51:16.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.159 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.159 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-13T03:51:16.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.160 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.160 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-04-13T03:51:16.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-04-13T03:51:16.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.160 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.161 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-04-13T03:51:16.161 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.161 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.161 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-13T03:51:16.161 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.161 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.162 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-04-13T03:51:16.162 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-04-13T03:51:16.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.162 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.162 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.163 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.163 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.163 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-04-13T03:51:16.163 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.163 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:16.163 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:16.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.164 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.164 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-04-13T03:51:16.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-04-13T03:51:16.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-13T03:51:16.164 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.165 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.166 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.166 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.166 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.166 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.166 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.166 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.167 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.167 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.167 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.168 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.169 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.170 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.171 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.171 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.171 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.171 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.171 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.171 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.171 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.171 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.172 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.172 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.172 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.172 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.172 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.172 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.173 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.173 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.173 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.173 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.173 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.174 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.174 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-04-13T03:51:16.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-04-13T03:51:16.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.174 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.175 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.175 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-04-13T03:51:16.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.175 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-04-13T03:51:16.176 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-04-13T03:51:16.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.176 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-04-13T03:51:16.176 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.176 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.176 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.176 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-04-13T03:51:16.177 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.177 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-13T03:51:16.177 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-13T03:51:16.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.177 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-04-13T03:51:16.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.178 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.178 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-13T03:51:16.178 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.178 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-04-13T03:51:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-04-13T03:51:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:16.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-04-13T03:51:16.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-13T03:51:16.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.180 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.180 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-04-13T03:51:16.180 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-04-13T03:51:16.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.181 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.181 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.181 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-04-13T03:51:16.181 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.182 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-04-13T03:51:16.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-04-13T03:51:16.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.182 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-04-13T03:51:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-13T03:51:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.183 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-04-13T03:51:16.184 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.184 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.184 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.184 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.185 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.185 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.187 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.187 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.187 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.187 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.188 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.188 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.188 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.188 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.189 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.189 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.189 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.190 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.190 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.190 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.190 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.190 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.190 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.191 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.191 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.191 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-04-13T03:51:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.194 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.194 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.194 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.194 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.194 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.194 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.194 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.194 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.194 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.195 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.195 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.195 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.195 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.196 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.198 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.198 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.198 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.198 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.198 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.198 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.198 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.198 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.199 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.199 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.199 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.200 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.200 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.200 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.200 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-04-13T03:51:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.201 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.202 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.202 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-04-13T03:51:16.202 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-04-13T03:51:16.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.202 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.202 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-04-13T03:51:16.202 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-13T03:51:16.203 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.203 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.203 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-04-13T03:51:16.203 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-04-13T03:51:16.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.204 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.204 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-04-13T03:51:16.204 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.204 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.204 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-13T03:51:16.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.205 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.205 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.205 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.205 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.205 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.206 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.206 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.206 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.206 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.206 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.206 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.207 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.207 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.207 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.207 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.208 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.208 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.208 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.208 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.208 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.208 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.208 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.210 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.210 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.210 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.210 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.210 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.210 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.211 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.211 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.211 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.211 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.211 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.212 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.212 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.212 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.212 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.213 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.213 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-04-13T03:51:16.213 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.213 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.213 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.214 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-04-13T03:51:16.214 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-04-13T03:51:16.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.214 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.214 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-04-13T03:51:16.214 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.214 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.215 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.215 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-04-13T03:51:16.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-13T03:51:16.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.215 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.216 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.216 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.216 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.216 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.217 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.217 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.217 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.217 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.217 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.217 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.217 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.217 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.217 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.218 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.219 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.220 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.220 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.220 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.220 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.220 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.221 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.221 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.221 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.221 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.221 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.221 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.221 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.222 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.222 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.222 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.222 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.222 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.223 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.223 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.223 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.223 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.223 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.223 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-04-13T03:51:16.223 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-04-13T03:51:16.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.224 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.224 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.224 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.224 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.224 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.224 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-04-13T03:51:16.224 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-13T03:51:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-04-13T03:51:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-04-13T03:51:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.225 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-04-13T03:51:16.226 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.226 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.226 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.226 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-04-13T03:51:16.226 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.226 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.227 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-04-13T03:51:16.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-04-13T03:51:16.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.227 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.227 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-04-13T03:51:16.227 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.227 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-04-13T03:51:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-04-13T03:51:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-04-13T03:51:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.228 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.230 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-04-13T03:51:16.230 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-04-13T03:51:16.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.230 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.230 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-04-13T03:51:16.230 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.231 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.231 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-13T03:51:16.231 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-13T03:51:16.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.231 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-04-13T03:51:16.232 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.232 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.232 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.232 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.232 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.232 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.233 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.233 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.233 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.233 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.233 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.233 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.234 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.234 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.234 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.234 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.234 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.234 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.235 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.235 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.235 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.235 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.235 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.236 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.236 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.236 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.237 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.237 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.237 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.237 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.237 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.238 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.238 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.238 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.238 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.238 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-04-13T03:51:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.241 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.241 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.241 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.241 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.241 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.241 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.241 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.242 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.242 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.242 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.242 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.242 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.242 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.242 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.242 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.243 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.243 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.243 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.243 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.245 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.245 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.245 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.245 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.245 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.246 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.247 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-04-13T03:51:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.249 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.250 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.251 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.252 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.252 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.252 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.252 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.253 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.254 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.254 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.254 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.254 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.254 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.255 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.255 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.255 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.255 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.255 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.255 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.255 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.255 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.257 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.257 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.257 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.257 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.257 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-13T03:51:16.258 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.258 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.258 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.258 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-04-13T03:51:16.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-04-13T03:51:16.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.259 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.259 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.259 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.259 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.259 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-13T03:51:16.259 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.260 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.260 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.260 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.260 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.261 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.261 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.261 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.261 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.261 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.261 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.261 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.261 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.262 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.262 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.262 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.262 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.263 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.263 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.263 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.263 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.263 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.264 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.264 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.264 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.264 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.264 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.265 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.265 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.265 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.265 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.265 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.265 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.265 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.266 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.266 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.266 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.266 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.266 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.267 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.267 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.267 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.267 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.267 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.268 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.268 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.268 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.268 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.268 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-04-13T03:51:16.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-13T03:51:16.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-13T03:51:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-04-13T03:51:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-04-13T03:51:16.270 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-04-13T03:51:16.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.270 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.270 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.270 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-04-13T03:51:16.271 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.271 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.271 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.271 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.272 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.272 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.272 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.272 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.272 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.272 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.273 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.274 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.274 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.274 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.274 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.275 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.275 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.275 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.275 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.275 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.275 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.277 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.277 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.277 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.277 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.278 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.279 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-04-13T03:51:16.279 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.280 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.280 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.280 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.280 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.281 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.281 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.281 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.281 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.281 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.281 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.281 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.281 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.281 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.282 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.282 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.282 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.283 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.283 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.283 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.283 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.283 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.283 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.283 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.284 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.285 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.286 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.286 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.286 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.286 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.287 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.287 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.287 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.287 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.287 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.287 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-04-13T03:51:16.288 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.288 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.288 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.288 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.290 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-04-13T03:51:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-04-13T03:51:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-13T03:51:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-04-13T03:51:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-04-13T03:51:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-13T03:51:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-04-13T03:51:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-04-13T03:51:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-04-13T03:51:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-13T03:51:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-13T03:51:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-13T03:51:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-04-13T03:51:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-04-13T03:51:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-04-13T03:51:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-04-13T03:51:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-04-13T03:51:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-04-13T03:51:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-04-13T03:51:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-13T03:51:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-13T03:51:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-04-13T03:51:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-04-13T03:51:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-13T03:51:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-13T03:51:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-04-13T03:51:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-04-13T03:51:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-04-13T03:51:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-04-13T03:51:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-13T03:51:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-13T03:51:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-13T03:51:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-13T03:51:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-13T03:51:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-13T03:51:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-04-13T03:51:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-13T03:51:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-04-13T03:51:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-04-13T03:51:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-04-13T03:51:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-13T03:51:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-13T03:51:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-04-13T03:51:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-04-13T03:51:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-04-13T03:51:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-13T03:51:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-13T03:51:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-13T03:51:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-13T03:51:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-13T03:51:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-13T03:51:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-04-13T03:51:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-04-13T03:51:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-04-13T03:51:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-04-13T03:51:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-04-13T03:51:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-04-13T03:51:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-04-13T03:51:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-04-13T03:51:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-04-13T03:51:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-13T03:51:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-13T03:51:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-13T03:51:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-13T03:51:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-04-13T03:51:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-04-13T03:51:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-04-13T03:51:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-04-13T03:51:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-04-13T03:51:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-13T03:51:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-04-13T03:51:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-04-13T03:51:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-04-13T03:51:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-04-13T03:51:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-04-13T03:51:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-04-13T03:51:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-04-13T03:51:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-04-13T03:51:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-04-13T03:51:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-04-13T03:51:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-04-13T03:51:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-04-13T03:51:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-04-13T03:51:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-04-13T03:51:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-04-13T03:51:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-04-13T03:51:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-04-13T03:51:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-04-13T03:51:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-04-13T03:51:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-04-13T03:51:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-04-13T03:51:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-04-13T03:51:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-04-13T03:51:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-04-13T03:51:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-04-13T03:51:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-13T03:51:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-04-13T03:51:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-04-13T03:51:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-04-13T03:51:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-04-13T03:51:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-04-13T03:51:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-04-13T03:51:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-04-13T03:51:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-04-13T03:51:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-04-13T03:51:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-04-13T03:51:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-04-13T03:51:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-13T03:51:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-04-13T03:51:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-04-13T03:51:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-04-13T03:51:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-04-13T03:51:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-13T03:51:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-04-13T03:51:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-04-13T03:51:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-13T03:51:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-04-13T03:51:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-04-13T03:51:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-13T03:51:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-04-13T03:51:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-04-13T03:51:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-13T03:51:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-04-13T03:51:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-04-13T03:51:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-13T03:51:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-04-13T03:51:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-04-13T03:51:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-04-13T03:51:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-04-13T03:51:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-13T03:51:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-13T03:51:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-13T03:51:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-04-13T03:51:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-04-13T03:51:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-04-13T03:51:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-04-13T03:51:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-04-13T03:51:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-04-13T03:51:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-04-13T03:51:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-04-13T03:51:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-04-13T03:51:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-04-13T03:51:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-04-13T03:51:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-04-13T03:51:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-04-13T03:51:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-04-13T03:51:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-04-13T03:51:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-13T03:51:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-04-13T03:51:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.500 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.500 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.500 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.500 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.500 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.500 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.501 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.501 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.501 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.501 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-13T03:51:16.502 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.502 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.502 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.502 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.502 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.502 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.503 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.503 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.503 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.503 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.503 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.503 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.504 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.505 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.505 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.505 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.505 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.505 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.505 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.505 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.506 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.506 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.506 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.506 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.506 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.506 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.507 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.507 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.507 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.507 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.507 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.507 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.507 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.507 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.507 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.508 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.508 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.508 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.508 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.509 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.509 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.509 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.509 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.509 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.510 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.510 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.510 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-04-13T03:51:16.510 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-04-13T03:51:16.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.510 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.512 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-04-13T03:51:16.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-04-13T03:51:16.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.512 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.513 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.513 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-04-13T03:51:16.513 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.513 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.513 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-04-13T03:51:16.513 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-04-13T03:51:16.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.514 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.514 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:16.514 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.514 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.514 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-13T03:51:16.514 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-04-13T03:51:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-04-13T03:51:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.516 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.516 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.516 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-13T03:51:16.516 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.516 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.516 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.517 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-04-13T03:51:16.517 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.517 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.517 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.517 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-04-13T03:51:16.517 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-04-13T03:51:16.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:16.517 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.518 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.518 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-04-13T03:51:16.518 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.518 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.518 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.518 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.519 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.519 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.519 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.519 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-04-13T03:51:16.519 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-04-13T03:51:16.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.520 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.520 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.520 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.520 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.520 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.521 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.521 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.521 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.521 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:16.521 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:16.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.522 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.522 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:16.522 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.522 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.523 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.523 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.523 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.523 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.523 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.524 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.524 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.524 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.525 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.525 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.525 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.525 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.526 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.526 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.526 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.526 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.526 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.526 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.526 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.527 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.527 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.527 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.527 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.527 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.528 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.528 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.528 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.528 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.529 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.529 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.529 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.529 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.529 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.529 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.530 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.531 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.531 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.531 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.531 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.532 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.532 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.532 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.532 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-04-13T03:51:16.532 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-04-13T03:51:16.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.533 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.533 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.533 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.533 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.533 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.533 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.533 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.534 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-04-13T03:51:16.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-04-13T03:51:16.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.534 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.534 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-04-13T03:51:16.534 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.534 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.534 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.535 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.535 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.535 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.535 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-04-13T03:51:16.535 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-04-13T03:51:16.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.535 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-13T03:51:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.536 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "gid": 25466, 2024-04-13T03:51:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-04-13T03:51:16.537 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_features": 4540701547738038271, 2024-04-13T03:51:16.538 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-04-13T03:51:16.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.538 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-04-13T03:51:16.538 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.538 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.538 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.538 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:16.538 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:16.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.539 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:16.539 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.539 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.539 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-04-13T03:51:16.539 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.540 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.540 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.540 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.540 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.541 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.541 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.541 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.541 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.541 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.541 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.542 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.542 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.542 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.542 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.542 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.543 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.543 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.544 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.544 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.544 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.544 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.545 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.545 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.545 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.545 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.545 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.545 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.545 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.545 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.545 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.546 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.546 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.546 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.546 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.546 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.547 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.548 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-04-13T03:51:16.548 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-04-13T03:51:16.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.548 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.548 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.548 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.549 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.549 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-04-13T03:51:16.549 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.549 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.549 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.549 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-04-13T03:51:16.549 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-04-13T03:51:16.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.550 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.550 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-04-13T03:51:16.550 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.550 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.550 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-04-13T03:51:16.550 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.551 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.551 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-04-13T03:51:16.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-04-13T03:51:16.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.551 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.551 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.552 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.552 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.552 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-04-13T03:51:16.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.552 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-04-13T03:51:16.552 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-04-13T03:51:16.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.553 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.553 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.553 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.553 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.553 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-04-13T03:51:16.554 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.554 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.554 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.554 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-04-13T03:51:16.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-04-13T03:51:16.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.554 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.555 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-04-13T03:51:16.555 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-04-13T03:51:16.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.555 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.556 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-04-13T03:51:16.556 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-04-13T03:51:16.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.556 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.556 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.557 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.557 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-04-13T03:51:16.557 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.557 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.557 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-04-13T03:51:16.557 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-04-13T03:51:16.557 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.558 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.558 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.558 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.558 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.558 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.558 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.558 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-13T03:51:16.558 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.559 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.559 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-04-13T03:51:16.559 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-04-13T03:51:16.559 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.559 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.560 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.560 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.560 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-04-13T03:51:16.560 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.560 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-04-13T03:51:16.561 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.561 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.561 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.561 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-04-13T03:51:16.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-04-13T03:51:16.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.562 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.562 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.562 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.562 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.562 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.562 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-04-13T03:51:16.562 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.563 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.563 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-04-13T03:51:16.563 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-04-13T03:51:16.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.563 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.563 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-04-13T03:51:16.563 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.564 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.564 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.564 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-04-13T03:51:16.564 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-13T03:51:16.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.564 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-04-13T03:51:16.564 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-04-13T03:51:16.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:16.565 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:16.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-13T03:51:16.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-13T03:51:16.565 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.565 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-13T03:51:16.565 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-13T03:51:16.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.566 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.566 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-04-13T03:51:16.566 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-04-13T03:51:16.566 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:16.566 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.566 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.566 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-04-13T03:51:16.567 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:16.567 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-04-13T03:51:16.567 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.567 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-13T03:51:16.567 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.567 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.567 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.567 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-13T03:51:16.568 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-13T03:51:16.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.569 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-04-13T03:51:16.569 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-04-13T03:51:16.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:16.569 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.569 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-04-13T03:51:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-04-13T03:51:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-04-13T03:51:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-04-13T03:51:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-13T03:51:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-04-13T03:51:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-04-13T03:51:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-04-13T03:51:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-04-13T03:51:16.572 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-13T03:51:16.572 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.572 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.572 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-04-13T03:51:16.572 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-04-13T03:51:16.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:16.572 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.573 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:16.573 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-13T03:51:16.573 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-13T03:51:16.573 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.573 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-13T03:51:16.573 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-13T03:51:16.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.573 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.574 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.574 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.574 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.574 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.574 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.574 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.574 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.575 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.575 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.575 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.575 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.575 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.575 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.575 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.575 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.575 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.576 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.576 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.576 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.576 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.576 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.576 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.576 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.578 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.578 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.578 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.578 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.578 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.578 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.579 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.579 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.579 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.579 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.579 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.579 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.579 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.579 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.579 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.580 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.580 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.580 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.580 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.580 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.580 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.580 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.581 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.581 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.581 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.581 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.581 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.581 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.581 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.582 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-04-13T03:51:16.582 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-04-13T03:51:16.582 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:16.582 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.582 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.582 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:16.582 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.582 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.583 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.583 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-13T03:51:16.583 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.583 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.583 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-13T03:51:16.583 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-13T03:51:16.584 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.584 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.584 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-04-13T03:51:16.584 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.584 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.584 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.584 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-04-13T03:51:16.584 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-04-13T03:51:16.585 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-04-13T03:51:16.585 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-04-13T03:51:16.585 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-04-13T03:51:16.585 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.585 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-04-13T03:51:16.585 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.585 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.585 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.586 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-04-13T03:51:16.586 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-04-13T03:51:16.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.586 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.586 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.586 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.586 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.586 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.587 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-13T03:51:16.587 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.587 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.587 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:16.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:16.587 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.587 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:16.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.588 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.588 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.588 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-13T03:51:16.588 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.588 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-04-13T03:51:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-04-13T03:51:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:16.589 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.590 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.590 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-13T03:51:16.590 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-13T03:51:16.590 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.590 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.590 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-04-13T03:51:16.590 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-04-13T03:51:16.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:16.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.591 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.591 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.591 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.591 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.591 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-13T03:51:16.591 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.592 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.592 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-04-13T03:51:16.592 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.593 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.594 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.594 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-13T03:51:16.594 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.594 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.594 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-04-13T03:51:16.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-04-13T03:51:16.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.595 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.595 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-04-13T03:51:16.595 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.595 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.595 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.595 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-13T03:51:16.596 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.596 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.596 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-04-13T03:51:16.596 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-04-13T03:51:16.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.596 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.596 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.597 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-04-13T03:51:16.597 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.597 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.597 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.597 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-13T03:51:16.597 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.597 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.597 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.598 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-04-13T03:51:16.598 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-04-13T03:51:16.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.598 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.598 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.598 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.598 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.599 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.599 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-13T03:51:16.599 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-13T03:51:16.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.599 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.599 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-04-13T03:51:16.599 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-04-13T03:51:16.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-13T03:51:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.601 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-04-13T03:51:16.601 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-04-13T03:51:16.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:16.601 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.601 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-04-13T03:51:16.601 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.602 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.602 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.602 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-13T03:51:16.602 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.602 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.602 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.602 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-04-13T03:51:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-04-13T03:51:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-04-13T03:51:16.604 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-04-13T03:51:16.604 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-04-13T03:51:16.604 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.604 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-13T03:51:16.604 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.604 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.604 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-04-13T03:51:16.605 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-04-13T03:51:16.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.605 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.605 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.605 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.605 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.605 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.605 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-13T03:51:16.606 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.606 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.606 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.606 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-04-13T03:51:16.606 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-04-13T03:51:16.606 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.606 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.607 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.607 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.607 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.607 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.607 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-13T03:51:16.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.607 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.608 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-04-13T03:51:16.608 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-04-13T03:51:16.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.608 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.608 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.608 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.608 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.608 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.609 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.609 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-13T03:51:16.609 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.609 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.609 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-04-13T03:51:16.609 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-04-13T03:51:16.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.610 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.610 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.610 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-13T03:51:16.610 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.610 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.610 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.610 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:16.610 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.610 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.611 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.611 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-04-13T03:51:16.611 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-04-13T03:51:16.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.611 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.611 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-13T03:51:16.611 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-04-13T03:51:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-04-13T03:51:16.612 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-04-13T03:51:16.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.613 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.613 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.613 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-13T03:51:16.613 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.613 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.613 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.614 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-04-13T03:51:16.614 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.614 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.614 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.614 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-04-13T03:51:16.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-04-13T03:51:16.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-13T03:51:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.615 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.616 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-04-13T03:51:16.617 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-04-13T03:51:16.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.617 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.618 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-13T03:51:16.618 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.618 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.618 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.618 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-04-13T03:51:16.618 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.618 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.619 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-04-13T03:51:16.619 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-04-13T03:51:16.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.619 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.619 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.619 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-13T03:51:16.619 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.619 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-04-13T03:51:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-04-13T03:51:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-04-13T03:51:16.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.621 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.621 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-13T03:51:16.621 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.621 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.621 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.621 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-04-13T03:51:16.621 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.622 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.622 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-04-13T03:51:16.622 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-04-13T03:51:16.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.622 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.622 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-13T03:51:16.623 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.623 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.623 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.623 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-04-13T03:51:16.623 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.623 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.623 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-04-13T03:51:16.624 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-04-13T03:51:16.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.624 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.624 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-13T03:51:16.624 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.624 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.624 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.625 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-04-13T03:51:16.625 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.625 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.625 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.625 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-04-13T03:51:16.625 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-04-13T03:51:16.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.625 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-13T03:51:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-04-13T03:51:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-04-13T03:51:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-04-13T03:51:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-13T03:51:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-04-13T03:51:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-04-13T03:51:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-13T03:51:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-04-13T03:51:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-04-13T03:51:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-04-13T03:51:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-13T03:51:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-04-13T03:51:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-04-13T03:51:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-13T03:51:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-04-13T03:51:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-04-13T03:51:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-04-13T03:51:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-13T03:51:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-04-13T03:51:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-04-13T03:51:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-04-13T03:51:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-13T03:51:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-04-13T03:51:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-04-13T03:51:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-04-13T03:51:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-04-13T03:51:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-04-13T03:51:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-04-13T03:51:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-13T03:51:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-04-13T03:51:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-04-13T03:51:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-04-13T03:51:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-13T03:51:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-04-13T03:51:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-04-13T03:51:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-04-13T03:51:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-13T03:51:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-04-13T03:51:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-04-13T03:51:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-04-13T03:51:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-04-13T03:51:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-04-13T03:51:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-04-13T03:51:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-13T03:51:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-04-13T03:51:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-04-13T03:51:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-04-13T03:51:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-04-13T03:51:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-04-13T03:51:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-13T03:51:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-04-13T03:51:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-04-13T03:51:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-04-13T03:51:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-13T03:51:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-04-13T03:51:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-04-13T03:51:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-13T03:51:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-04-13T03:51:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-04-13T03:51:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-13T03:51:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-13T03:51:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-13T03:51:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-13T03:51:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-04-13T03:51:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-13T03:51:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-04-13T03:51:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-04-13T03:51:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-04-13T03:51:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-13T03:51:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-04-13T03:51:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-04-13T03:51:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-04-13T03:51:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-13T03:51:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-13T03:51:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-04-13T03:51:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-04-13T03:51:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-04-13T03:51:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-04-13T03:51:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-04-13T03:51:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-04-13T03:51:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-04-13T03:51:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-13T03:51:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-04-13T03:51:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-04-13T03:51:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-13T03:51:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-13T03:51:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-04-13T03:51:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-04-13T03:51:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-13T03:51:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-04-13T03:51:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-04-13T03:51:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-13T03:51:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-04-13T03:51:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-04-13T03:51:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-13T03:51:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-04-13T03:51:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-04-13T03:51:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-13T03:51:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-04-13T03:51:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-04-13T03:51:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-13T03:51:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-04-13T03:51:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-04-13T03:51:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-04-13T03:51:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-04-13T03:51:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-04-13T03:51:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-04-13T03:51:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-04-13T03:51:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-04-13T03:51:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-13T03:51:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-04-13T03:51:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-04-13T03:51:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-13T03:51:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-04-13T03:51:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-04-13T03:51:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-04-13T03:51:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-04-13T03:51:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-04-13T03:51:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-13T03:51:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-04-13T03:51:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-04-13T03:51:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-13T03:51:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-04-13T03:51:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-04-13T03:51:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-04-13T03:51:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-04-13T03:51:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-13T03:51:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-04-13T03:51:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-04-13T03:51:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-04-13T03:51:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-13T03:51:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-04-13T03:51:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-13T03:51:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-13T03:51:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-04-13T03:51:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-13T03:51:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-13T03:51:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-13T03:51:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-04-13T03:51:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-04-13T03:51:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-13T03:51:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-13T03:51:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-04-13T03:51:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-13T03:51:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-13T03:51:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-13T03:51:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-13T03:51:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-13T03:51:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-13T03:51:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-13T03:51:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-13T03:51:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-13T03:51:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-13T03:51:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-13T03:51:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-13T03:51:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-13T03:51:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-13T03:51:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-13T03:51:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-13T03:51:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-13T03:51:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-04-13T03:51:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-13T03:51:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-13T03:51:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-13T03:51:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-13T03:51:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-04-13T03:51:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-04-13T03:51:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-04-13T03:51:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-13T03:51:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-13T03:51:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-13T03:51:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-13T03:51:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-13T03:51:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-13T03:51:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-13T03:51:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-13T03:51:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-13T03:51:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-13T03:51:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-13T03:51:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-13T03:51:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "[]", 2024-04-13T03:51:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "MULTICLUSTER_CONFIG": { 2024-04-13T03:51:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-13T03:51:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "{}", 2024-04-13T03:51:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-04-13T03:51:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-13T03:51:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-13T03:51:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-13T03:51:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-13T03:51:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-13T03:51:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-13T03:51:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-13T03:51:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-13T03:51:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-13T03:51:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-13T03:51:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-13T03:51:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-13T03:51:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-13T03:51:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-13T03:51:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-13T03:51:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-13T03:51:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-13T03:51:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-04-13T03:51:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-13T03:51:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-13T03:51:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-13T03:51:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-13T03:51:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-13T03:51:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-13T03:51:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-13T03:51:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-13T03:51:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-04-13T03:51:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-13T03:51:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-13T03:51:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-04-13T03:51:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-04-13T03:51:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-13T03:51:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-13T03:51:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-13T03:51:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-13T03:51:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-04-13T03:51:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-13T03:51:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-04-13T03:51:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-13T03:51:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-04-13T03:51:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-13T03:51:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-13T03:51:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-13T03:51:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-13T03:51:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-13T03:51:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-13T03:51:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-13T03:51:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-04-13T03:51:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-04-13T03:51:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-04-13T03:51:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-04-13T03:51:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-04-13T03:51:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-04-13T03:51:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-04-13T03:51:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-04-13T03:51:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-04-13T03:51:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-04-13T03:51:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-13T03:51:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-13T03:51:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-04-13T03:51:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-04-13T03:51:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-04-13T03:51:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-04-13T03:51:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-04-13T03:51:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-13T03:51:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-13T03:51:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-13T03:51:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-13T03:51:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-13T03:51:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-04-13T03:51:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-13T03:51:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-13T03:51:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-04-13T03:51:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-04-13T03:51:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-04-13T03:51:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-13T03:51:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-13T03:51:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-04-13T03:51:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-04-13T03:51:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-13T03:51:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-13T03:51:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-13T03:51:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-13T03:51:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-13T03:51:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-04-13T03:51:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-04-13T03:51:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-13T03:51:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-04-13T03:51:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-04-13T03:51:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-04-13T03:51:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-04-13T03:51:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-04-13T03:51:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-04-13T03:51:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-13T03:51:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-04-13T03:51:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-04-13T03:51:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-04-13T03:51:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-13T03:51:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-04-13T03:51:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-04-13T03:51:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-04-13T03:51:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-04-13T03:51:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-04-13T03:51:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-04-13T03:51:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-13T03:51:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-13T03:51:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-04-13T03:51:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-04-13T03:51:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-13T03:51:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-04-13T03:51:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-04-13T03:51:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-04-13T03:51:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-04-13T03:51:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-13T03:51:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-04-13T03:51:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-04-13T03:51:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-04-13T03:51:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-13T03:51:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-04-13T03:51:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-04-13T03:51:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-04-13T03:51:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-04-13T03:51:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-04-13T03:51:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-04-13T03:51:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-04-13T03:51:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-04-13T03:51:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-04-13T03:51:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-13T03:51:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-04-13T03:51:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-04-13T03:51:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-04-13T03:51:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-13T03:51:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-04-13T03:51:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-04-13T03:51:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.855 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-04-13T03:51:16.855 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.855 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.855 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:16.855 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:16.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.856 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.856 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-04-13T03:51:16.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-04-13T03:51:16.856 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.856 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.856 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-13T03:51:16.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.857 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.857 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.857 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.857 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.858 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.859 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.860 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.860 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.860 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.860 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.860 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.860 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.862 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.862 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.862 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.862 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.862 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.862 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.862 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.862 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.864 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.864 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.864 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.864 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.864 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.864 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.864 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.865 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.865 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-04-13T03:51:16.865 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-04-13T03:51:16.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.865 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.866 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.866 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-04-13T03:51:16.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.866 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-04-13T03:51:16.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-04-13T03:51:16.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-04-13T03:51:16.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.867 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-04-13T03:51:16.867 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-13T03:51:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-13T03:51:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.868 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-04-13T03:51:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-13T03:51:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-04-13T03:51:16.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-04-13T03:51:16.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.870 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.870 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:16.870 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:16.870 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-04-13T03:51:16.870 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-13T03:51:16.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.871 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.871 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-04-13T03:51:16.871 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-04-13T03:51:16.871 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.871 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.872 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.872 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.872 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.872 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.872 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-04-13T03:51:16.872 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.872 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.873 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-04-13T03:51:16.873 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-04-13T03:51:16.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.873 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.873 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.873 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-04-13T03:51:16.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-13T03:51:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-04-13T03:51:16.875 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.875 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.875 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.875 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.876 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.876 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.876 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.876 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.876 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.876 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.877 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.878 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.878 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.878 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.878 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.878 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.879 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.879 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.880 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.880 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.880 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.880 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.880 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.881 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.881 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.881 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.881 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.881 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.881 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.882 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.882 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.882 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.882 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.882 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.883 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.885 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.885 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.885 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.885 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.885 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.885 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.885 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.885 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.885 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.886 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.886 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.886 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.886 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.887 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.888 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.888 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.889 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.889 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.889 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.889 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.889 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.889 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.889 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.889 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.890 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.890 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.890 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.890 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.891 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.891 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.891 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.891 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-04-13T03:51:16.892 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.892 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.892 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.892 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-04-13T03:51:16.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-04-13T03:51:16.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.893 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-04-13T03:51:16.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.893 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-13T03:51:16.893 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.894 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.894 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-04-13T03:51:16.894 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-04-13T03:51:16.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.894 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.894 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:16.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-04-13T03:51:16.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.895 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.895 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-13T03:51:16.895 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.895 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.895 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.896 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.896 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.896 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.896 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.896 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.897 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.897 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.897 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.897 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.897 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.897 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.897 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.897 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.898 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.898 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.898 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.898 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.898 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.899 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.899 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.899 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.900 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.901 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.901 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.901 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.901 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.901 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.901 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.903 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-04-13T03:51:16.904 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.904 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.904 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.904 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-04-13T03:51:16.904 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-04-13T03:51:16.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.904 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.906 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.906 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.906 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.906 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.906 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.906 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.907 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.907 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.907 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.907 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.907 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.907 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.907 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.907 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.907 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.908 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.908 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.908 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.908 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.910 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.911 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.911 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.911 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.911 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.911 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.911 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.911 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.911 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.912 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.912 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.912 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.912 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.912 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.912 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.913 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.913 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.913 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.913 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.913 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.913 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.913 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.914 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-04-13T03:51:16.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-04-13T03:51:16.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.914 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.914 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.914 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.915 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.915 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.915 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-04-13T03:51:16.915 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-13T03:51:16.915 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.915 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.915 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-04-13T03:51:16.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-04-13T03:51:16.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.916 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.916 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-04-13T03:51:16.916 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.916 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.916 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.916 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-04-13T03:51:16.917 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.917 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.917 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-04-13T03:51:16.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-04-13T03:51:16.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:16.917 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-04-13T03:51:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-04-13T03:51:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.919 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-04-13T03:51:16.919 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-04-13T03:51:16.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.919 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.919 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.919 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.919 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.920 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.920 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-04-13T03:51:16.920 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.920 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.920 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-04-13T03:51:16.920 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-04-13T03:51:16.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-04-13T03:51:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-13T03:51:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-13T03:51:16.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.922 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.922 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-04-13T03:51:16.922 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.922 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.923 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.923 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.923 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.923 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.923 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.923 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.924 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.924 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.924 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.924 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.924 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.924 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.924 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.924 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.924 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.925 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.926 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.927 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.927 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.927 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.927 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.927 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.927 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.927 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.928 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.928 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.928 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.928 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.928 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.928 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.928 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.928 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.929 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.929 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.929 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.929 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.929 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.930 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.930 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.930 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.930 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.930 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.930 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.930 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.931 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.932 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.932 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.932 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.932 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.932 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.932 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.932 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.933 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.934 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.934 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.934 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.934 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.934 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.934 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.934 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.935 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.935 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.935 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.935 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.935 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.936 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.936 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.936 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.936 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.936 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.936 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.936 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.936 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.936 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.937 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.937 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.937 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.937 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.937 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.937 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.937 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.938 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.938 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.938 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.938 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.938 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.938 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.938 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.939 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.939 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.939 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-04-13T03:51:16.939 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.939 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.940 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.940 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.940 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.940 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.940 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.940 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.940 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.941 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.941 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.941 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.941 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.941 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.941 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.941 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.941 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.941 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.942 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.942 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.942 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.942 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.942 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.944 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.944 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.944 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.944 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.944 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.944 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.944 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.945 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.946 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.946 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.946 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.946 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.946 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.946 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.947 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.947 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.947 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.947 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.947 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.947 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.947 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.947 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-13T03:51:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-04-13T03:51:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-04-13T03:51:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.949 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.949 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.949 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.949 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.949 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.949 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.949 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-13T03:51:16.949 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.950 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.950 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.950 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.950 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.950 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.950 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.951 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.952 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.953 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.953 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.953 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.953 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.953 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.953 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.954 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.954 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.954 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.954 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.954 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.954 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.956 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.956 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.956 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.956 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.956 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.956 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.957 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.957 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.957 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.957 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.957 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.957 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.957 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.957 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-04-13T03:51:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-13T03:51:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.959 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.959 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-13T03:51:16.959 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-04-13T03:51:16.959 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-04-13T03:51:16.959 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.959 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-04-13T03:51:16.959 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.959 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.960 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-04-13T03:51:16.960 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.960 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.960 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.960 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.961 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.961 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.961 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.961 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.961 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.961 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.962 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.962 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.962 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.962 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.962 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.962 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.962 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.962 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.963 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.963 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.963 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.963 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.963 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.963 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.964 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.964 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.964 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.964 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.964 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.964 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.965 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.965 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.965 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.965 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.965 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.965 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.965 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.966 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.966 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.966 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.966 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.966 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.966 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.966 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.966 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.966 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.967 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.967 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.967 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.967 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.967 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.967 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.967 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.968 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.968 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.968 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.968 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.968 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-04-13T03:51:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.969 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.970 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.972 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.972 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.972 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.972 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.972 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.972 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.972 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.973 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.973 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.973 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.973 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.973 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.973 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.974 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.974 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.974 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.974 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.974 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.974 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.974 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.974 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.975 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.975 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.975 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.975 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.975 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.975 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.975 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.976 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.976 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.976 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.976 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.976 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.976 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.976 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.976 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.977 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.977 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.977 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.977 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-04-13T03:51:16.977 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.978 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.978 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.978 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.978 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.978 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.978 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.978 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.979 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.979 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.979 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.979 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.979 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.979 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.979 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.979 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.979 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.980 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.980 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.980 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.980 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.980 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.980 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.981 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.981 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.981 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.981 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.981 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.981 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.981 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.982 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.982 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.982 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.982 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.982 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.982 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.983 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.983 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.983 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.983 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.983 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.983 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.983 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.983 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.983 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.984 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.985 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.985 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.985 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.985 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.985 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.985 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.985 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:16.986 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:16.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:16.986 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.986 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:16.986 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.986 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.986 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.986 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.987 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.987 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.987 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-04-13T03:51:16.987 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-04-13T03:51:16.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:16.987 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.988 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-13T03:51:16.988 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-04-13T03:51:16.988 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.988 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.988 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-04-13T03:51:16.988 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-13T03:51:16.988 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.988 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:16.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:16.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-04-13T03:51:16.989 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:16.989 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:16.989 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:16.989 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-04-13T03:51:16.989 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-04-13T03:51:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-13T03:51:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.991 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.991 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-13T03:51:16.991 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-13T03:51:16.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.991 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:16.991 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:16.992 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.992 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.992 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.992 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.992 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.992 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.993 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:16.993 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:16.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.993 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.993 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:16.993 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.993 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.993 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.994 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.994 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.994 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.994 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.994 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.994 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.994 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.994 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.994 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.995 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.996 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.996 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:16.996 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.996 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.996 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.996 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:16.997 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:16.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:16.997 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.997 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:16.997 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:16.997 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:16.997 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:16.997 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:16.998 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:16.998 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:16.998 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:16.998 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:16.998 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:16.998 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:16.998 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:16.998 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:16.998 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:16.999 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:16.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:16.999 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:16.999 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:16.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:16.999 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:16.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:16.999 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:16.999 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.000 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.000 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.000 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.000 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.000 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-04-13T03:51:17.000 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.002 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.002 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:17.002 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:17.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.002 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.002 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:17.003 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.003 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.003 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.003 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-04-13T03:51:17.003 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.003 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-04-13T03:51:17.004 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.004 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.004 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.004 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-04-13T03:51:17.004 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-04-13T03:51:17.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.005 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.005 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.005 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.005 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.005 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.005 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.005 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.006 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.006 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-04-13T03:51:17.006 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-04-13T03:51:17.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.006 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.006 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.007 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.007 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.007 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.007 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-13T03:51:17.007 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-13T03:51:17.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.007 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.007 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.008 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.009 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.009 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.009 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.009 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.009 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.009 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.009 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.009 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.010 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.010 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.010 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.010 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.010 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.010 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.011 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.011 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.011 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.011 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.011 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.011 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.011 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.013 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.013 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.013 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.013 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.013 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.013 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.013 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.013 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.013 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.014 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-04-13T03:51:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-04-13T03:51:17.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.016 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.016 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.016 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.016 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.016 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.016 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.016 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-13T03:51:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-13T03:51:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.017 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-04-13T03:51:17.018 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.018 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.018 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.018 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.018 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.018 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.018 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-04-13T03:51:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-04-13T03:51:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-04-13T03:51:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.019 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.020 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.020 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.020 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.020 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-13T03:51:17.020 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-13T03:51:17.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.020 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.021 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.021 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-13T03:51:17.021 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.021 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.021 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.021 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-13T03:51:17.021 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.022 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.022 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-13T03:51:17.022 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-13T03:51:17.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.022 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.022 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-04-13T03:51:17.022 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.023 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.023 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.023 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-13T03:51:17.023 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.023 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.023 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-04-13T03:51:17.023 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-04-13T03:51:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-04-13T03:51:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-13T03:51:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-13T03:51:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.025 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-04-13T03:51:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-04-13T03:51:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-04-13T03:51:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-13T03:51:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-13T03:51:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-13T03:51:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-13T03:51:17.028 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-13T03:51:17.029 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.029 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.029 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-13T03:51:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.030 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.031 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.031 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.031 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.031 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.031 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.031 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.031 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.032 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.032 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.032 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.032 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.032 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.032 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.032 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.032 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.032 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.033 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.034 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.035 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.035 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.035 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.035 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.035 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.035 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.035 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.035 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.037 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.037 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.037 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.037 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.037 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.037 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.038 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.038 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.038 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.038 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.038 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb7900a3da0 msgr2=0x7fb7900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb7900a3da0 0x7fb7900a6260 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac102240 tx=0x7fb794005f90 comp rx=0 tx=0).stop 2024-04-13T03:51:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac0764f0 msgr2=0x7fb7ac19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac0764f0 0x7fb7ac19ef10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00e9f0 tx=0x7fb79c00eeb0 comp rx=0 tx=0).stop 2024-04-13T03:51:17.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 shutdown_connections 2024-04-13T03:51:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb7900a3da0 0x7fb7900a6260 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7ac112e30 0x7fb7ac1a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ac076e10 0x7fb7ac19f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 --2- 172.21.15.16:0/2235974691 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7ac0764f0 0x7fb7ac19ef10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 >> 172.21.15.16:0/2235974691 conn(0x7fb7ac0ffcc0 msgr2=0x7fb7ac114520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.611+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 shutdown_connections 2024-04-13T03:51:17.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:15.612+0000 7fb7b2400640 1 -- 172.21.15.16:0/2235974691 wait complete. 2024-04-13T03:51:17.043 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-04-13T03:51:17.043 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-04-13T03:51:17.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.043 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.044 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:17.044 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.044 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.044 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.044 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.044 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.044 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.045 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-04-13T03:51:17.045 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-04-13T03:51:17.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.045 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.045 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.045 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.046 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.046 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.046 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.046 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.046 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.046 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.046 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-04-13T03:51:17.046 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-04-13T03:51:17.047 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.047 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.047 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.047 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.047 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.047 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.047 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.047 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.047 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.048 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.048 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.048 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-04-13T03:51:17.048 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.048 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-04-13T03:51:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-04-13T03:51:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.049 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.050 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.050 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.050 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.050 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.050 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-13T03:51:17.050 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-13T03:51:17.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.051 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.051 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.051 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.051 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.051 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.051 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.051 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.052 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.052 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.052 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.052 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.052 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.052 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.053 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.053 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.053 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.053 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.053 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.053 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.053 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.053 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.053 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.054 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.054 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.054 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.054 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.054 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.054 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.055 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.055 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.055 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.055 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.055 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.055 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.055 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.055 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.056 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.056 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.056 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.056 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.056 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.056 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.056 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.057 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.057 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.057 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.057 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.057 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.057 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.057 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.057 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.057 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.058 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.058 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.058 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.058 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.058 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.058 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.058 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.058 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.059 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.059 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.059 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.059 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.059 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.059 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.059 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.059 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.059 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.060 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-13T03:51:17.060 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-13T03:51:17.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.060 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.060 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.060 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.060 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.061 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.061 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.061 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.061 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.061 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-13T03:51:17.061 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-13T03:51:17.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.062 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.062 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.062 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.062 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.062 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.062 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.062 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.063 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.063 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-04-13T03:51:17.063 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.063 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.064 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.064 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.064 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.064 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.064 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.064 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.064 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.064 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.065 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.065 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.065 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.065 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.065 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.065 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.065 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.065 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.066 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.066 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.066 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.066 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.066 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.066 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.066 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.066 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.067 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.067 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.067 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.067 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.067 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.067 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.067 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.068 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.068 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.068 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.068 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.068 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.068 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.068 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.069 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.069 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.069 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.069 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.069 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.069 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.069 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.069 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.069 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.070 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.070 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.070 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.070 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.070 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.070 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.071 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.071 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.071 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.071 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.071 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.071 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.071 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.071 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.071 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.072 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.072 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-04-13T03:51:17.072 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.072 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.073 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.073 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.073 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.073 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.073 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.073 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.073 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.074 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.075 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.075 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.075 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.075 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.075 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.075 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.076 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.076 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.076 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.076 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.076 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.076 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.076 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.077 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.077 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.077 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.077 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.077 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.077 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.077 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.077 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.077 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.078 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.078 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.078 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.078 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.078 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.078 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.078 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.078 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.078 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.079 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.079 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.079 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.079 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.079 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-04-13T03:51:17.081 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-04-13T03:51:17.081 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.081 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.081 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.081 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-04-13T03:51:17.081 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.081 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.081 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.081 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-13T03:51:17.082 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.082 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.082 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.082 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.082 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.082 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-04-13T03:51:17.082 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.083 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.083 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.083 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.083 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.083 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.083 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.083 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.083 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.084 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.084 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.084 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.084 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.084 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.084 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.084 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.084 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.084 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.085 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.086 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.087 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.088 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.089 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.090 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-04-13T03:51:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-04-13T03:51:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.091 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-04-13T03:51:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-04-13T03:51:17.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-04-13T03:51:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-04-13T03:51:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-04-13T03:51:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-04-13T03:51:17.095 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-04-13T03:51:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.096 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-04-13T03:51:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-04-13T03:51:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.097 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.098 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-04-13T03:51:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-04-13T03:51:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.099 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-04-13T03:51:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-04-13T03:51:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.100 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.102 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-04-13T03:51:17.102 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-04-13T03:51:17.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.102 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.102 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.102 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.102 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.102 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.103 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.103 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.103 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.103 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-04-13T03:51:17.103 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-04-13T03:51:17.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.103 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.104 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.104 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:17.104 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-04-13T03:51:17.104 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.104 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.104 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.105 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.105 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-04-13T03:51:17.105 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-04-13T03:51:17.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.105 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.105 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.105 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.106 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.106 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.106 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.106 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.106 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.106 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-04-13T03:51:17.106 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-04-13T03:51:17.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.107 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.107 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.107 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.107 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.107 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.107 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.107 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.108 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.108 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-04-13T03:51:17.108 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-04-13T03:51:17.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.108 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.108 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.109 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.109 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.109 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.109 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.109 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.109 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.109 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.110 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-13T03:51:17.110 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.110 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.110 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.110 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-04-13T03:51:17.110 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-04-13T03:51:17.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.111 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.111 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.111 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.111 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.111 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.111 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-04-13T03:51:17.111 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.112 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.112 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-04-13T03:51:17.112 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-04-13T03:51:17.112 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.112 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.112 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.112 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.113 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.113 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.113 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.113 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-04-13T03:51:17.113 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.113 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.113 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.114 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.114 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.114 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.114 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.114 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.114 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.114 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.115 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.115 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.115 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.115 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.115 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.115 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.115 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.115 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.116 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.116 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.116 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.116 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.116 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.116 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.117 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.117 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.117 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.117 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.117 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.117 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.117 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.118 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.118 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.118 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.118 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.118 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.118 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.118 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.119 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.119 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.119 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.119 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.119 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.119 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.119 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.119 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.120 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.120 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.120 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.120 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.120 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.120 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.120 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-04-13T03:51:17.122 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.122 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.122 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.122 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.122 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.122 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.123 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.125 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.125 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.125 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.125 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.125 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.125 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.125 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.126 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.126 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.126 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.126 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.126 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.126 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.127 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.127 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.127 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.127 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.127 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.127 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.127 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.127 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.127 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.128 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.128 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.128 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.128 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.128 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.128 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.128 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.129 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.129 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.129 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.129 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.129 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.129 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.129 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.130 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.130 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.130 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-04-13T03:51:17.130 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.130 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.130 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.130 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.131 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.131 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.131 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.131 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.131 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.131 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.132 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.132 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.132 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.132 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.132 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.132 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.132 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.132 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.132 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.133 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.133 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.133 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.133 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.133 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.133 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.133 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.134 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.135 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.135 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.135 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.135 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.135 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.135 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.135 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.136 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.136 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.136 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.136 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.136 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.136 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.136 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.136 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.137 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.137 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.137 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.137 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.137 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.137 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.138 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.138 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.138 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.138 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.138 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.138 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.138 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-04-13T03:51:17.139 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.139 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.139 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.139 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-04-13T03:51:17.139 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-04-13T03:51:17.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.139 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-13T03:51:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.141 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:17.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:17.141 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.141 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.141 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.141 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-04-13T03:51:17.141 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.141 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.142 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.142 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.142 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.142 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.142 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.142 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.144 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.144 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.144 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.144 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.144 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.144 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.144 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.145 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.145 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.145 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.145 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.145 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.145 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.145 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.146 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.146 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.146 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.146 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.146 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.146 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.147 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.147 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.147 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.147 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.147 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.147 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.147 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.147 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.148 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.148 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.148 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.148 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.148 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.148 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.148 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.149 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.149 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.149 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.149 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.149 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.149 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.149 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.150 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.150 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.150 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-04-13T03:51:17.150 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-04-13T03:51:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-04-13T03:51:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.151 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-13T03:51:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-04-13T03:51:17.152 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-04-13T03:51:17.153 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.153 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.153 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.153 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.153 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.153 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.153 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.153 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-13T03:51:17.154 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.154 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.154 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.154 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-04-13T03:51:17.154 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-04-13T03:51:17.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.154 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.155 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.155 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.155 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.155 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.155 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-13T03:51:17.155 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.155 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.156 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-04-13T03:51:17.156 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-04-13T03:51:17.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.156 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.156 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.156 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.156 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.157 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.157 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-13T03:51:17.157 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.157 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.157 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-04-13T03:51:17.157 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-04-13T03:51:17.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-13T03:51:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-04-13T03:51:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-04-13T03:51:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.159 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-04-13T03:51:17.160 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-04-13T03:51:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.161 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-04-13T03:51:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-04-13T03:51:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.162 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-13T03:51:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.163 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-13T03:51:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-13T03:51:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.164 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-04-13T03:51:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-04-13T03:51:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-04-13T03:51:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-04-13T03:51:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-04-13T03:51:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.167 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-04-13T03:51:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-04-13T03:51:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.168 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.169 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-04-13T03:51:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-04-13T03:51:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.170 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.172 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.174 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-04-13T03:51:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-04-13T03:51:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.179 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.180 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-04-13T03:51:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-04-13T03:51:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.181 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-04-13T03:51:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-04-13T03:51:17.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-13T03:51:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.183 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-04-13T03:51:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.184 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.185 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.186 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.187 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.188 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.189 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.191 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-13T03:51:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.193 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.194 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.195 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.196 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.197 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.198 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.199 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.200 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-04-13T03:51:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-04-13T03:51:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.201 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-04-13T03:51:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-13T03:51:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.202 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-04-13T03:51:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-04-13T03:51:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.203 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-04-13T03:51:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-04-13T03:51:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-04-13T03:51:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-13T03:51:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.205 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-04-13T03:51:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-04-13T03:51:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.206 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-13T03:51:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-04-13T03:51:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-04-13T03:51:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-04-13T03:51:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-04-13T03:51:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-04-13T03:51:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-04-13T03:51:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.210 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.211 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.213 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.214 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.217 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.218 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-04-13T03:51:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-04-13T03:51:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.221 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-04-13T03:51:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-04-13T03:51:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.222 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-04-13T03:51:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.223 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-04-13T03:51:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-04-13T03:51:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-13T03:51:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.224 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.225 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "modules": [ 2024-04-13T03:51:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-13T03:51:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard", 2024-04-13T03:51:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "iostat", 2024-04-13T03:51:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "nfs", 2024-04-13T03:51:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "restful", 2024-04-13T03:51:17.226 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_schedule" 2024-04-13T03:51:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "available_modules": [ 2024-04-13T03:51:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "alerts", 2024-04-13T03:51:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:17.227 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:17.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How frequently to reexamine health status", 2024-04-13T03:51:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.230 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.231 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.232 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.233 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.234 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.235 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.237 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_destination": { 2024-04-13T03:51:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_destination", 2024-04-13T03:51:17.238 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email address to send alerts to", 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.239 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_from_name": { 2024-04-13T03:51:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_from_name", 2024-04-13T03:51:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "Ceph", 2024-04-13T03:51:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.240 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Email From: name", 2024-04-13T03:51:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_host": { 2024-04-13T03:51:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_host", 2024-04-13T03:51:17.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP server", 2024-04-13T03:51:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_password": { 2024-04-13T03:51:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_password", 2024-04-13T03:51:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.243 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Password to authenticate with", 2024-04-13T03:51:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.244 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_port": { 2024-04-13T03:51:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_port", 2024-04-13T03:51:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "465", 2024-04-13T03:51:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.245 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP port", 2024-04-13T03:51:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_sender": { 2024-04-13T03:51:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_sender", 2024-04-13T03:51:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SMTP envelope sender", 2024-04-13T03:51:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_ssl": { 2024-04-13T03:51:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_ssl", 2024-04-13T03:51:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.248 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-13T03:51:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "smtp_user": { 2024-04-13T03:51:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "smtp_user", 2024-04-13T03:51:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.249 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "User to authenticate as", 2024-04-13T03:51:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.250 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "balancer", 2024-04-13T03:51:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.251 INFO:teuthology.orchestra.run.smithi016.stdout: "active": { 2024-04-13T03:51:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "active", 2024-04-13T03:51:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.252 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically balance PGs across cluster", 2024-04-13T03:51:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_time": { 2024-04-13T03:51:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_time", 2024-04-13T03:51:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0000", 2024-04-13T03:51:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "beginning time of day to automatically balance", 2024-04-13T03:51:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-13T03:51:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "begin_weekday": { 2024-04-13T03:51:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "begin_weekday", 2024-04-13T03:51:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-13T03:51:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-13T03:51:17.255 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-13T03:51:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-13T03:51:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_max_iterations": { 2024-04-13T03:51:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_max_iterations", 2024-04-13T03:51:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:17.256 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "25", 2024-04-13T03:51:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "250", 2024-04-13T03:51:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-13T03:51:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.257 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_metrics": { 2024-04-13T03:51:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_metrics", 2024-04-13T03:51:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "pgs,objects,bytes", 2024-04-13T03:51:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.258 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-13T03:51:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-13T03:51:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_compat_step": { 2024-04-13T03:51:17.259 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crush_compat_step", 2024-04-13T03:51:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.5", 2024-04-13T03:51:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0.001", 2024-04-13T03:51:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "0.999", 2024-04-13T03:51:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "aggressiveness of optimization", 2024-04-13T03:51:17.260 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-13T03:51:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "end_time": { 2024-04-13T03:51:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_time", 2024-04-13T03:51:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.261 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2359", 2024-04-13T03:51:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ending time of day to automatically balance", 2024-04-13T03:51:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-13T03:51:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "end_weekday": { 2024-04-13T03:51:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "end_weekday", 2024-04-13T03:51:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "0", 2024-04-13T03:51:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "6", 2024-04-13T03:51:17.263 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-13T03:51:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-13T03:51:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.265 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.266 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.267 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.268 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.269 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.271 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "min_score": { 2024-04-13T03:51:17.272 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_score", 2024-04-13T03:51:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-13T03:51:17.273 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-13T03:51:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-13T03:51:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.274 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "upmap", 2024-04-13T03:51:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-compat", 2024-04-13T03:51:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: "none", 2024-04-13T03:51:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: "read", 2024-04-13T03:51:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap", 2024-04-13T03:51:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap-read" 2024-04-13T03:51:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Balancer mode", 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_ids": { 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_ids", 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-13T03:51:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-13T03:51:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_deviation": { 2024-04-13T03:51:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_deviation", 2024-04-13T03:51:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-13T03:51:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-13T03:51:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "upmap_max_optimizations": { 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "upmap_max_optimizations", 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-13T03:51:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm", 2024-04-13T03:51:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_down_multiplier": { 2024-04-13T03:51:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_down_multiplier", 2024-04-13T03:51:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-13T03:51:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-13T03:51:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_refresh_rate": { 2024-04-13T03:51:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_refresh_rate", 2024-04-13T03:51:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "20", 2024-04-13T03:51:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-13T03:51:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "agent_starting_port": { 2024-04-13T03:51:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "agent_starting_port", 2024-04-13T03:51:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4721", 2024-04-13T03:51:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-13T03:51:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_ptrace": { 2024-04-13T03:51:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_ptrace", 2024-04-13T03:51:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-13T03:51:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-13T03:51:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_interval": { 2024-04-13T03:51:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_interval", 2024-04-13T03:51:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-13T03:51:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "autotune_memory_target_ratio": { 2024-04-13T03:51:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "autotune_memory_target_ratio", 2024-04-13T03:51:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0.7", 2024-04-13T03:51:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-13T03:51:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_log_destination": { 2024-04-13T03:51:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cephadm_log_destination", 2024-04-13T03:51:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "file", 2024-04-13T03:51:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "file,syslog", 2024-04-13T03:51:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "syslog" 2024-04-13T03:51:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-13T03:51:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "cgroups_split": { 2024-04-13T03:51:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cgroups_split", 2024-04-13T03:51:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-13T03:51:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "config_checks_enabled": { 2024-04-13T03:51:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_checks_enabled", 2024-04-13T03:51:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-13T03:51:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "config_dashboard": { 2024-04-13T03:51:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_dashboard", 2024-04-13T03:51:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-13T03:51:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_alertmanager": { 2024-04-13T03:51:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_alertmanager", 2024-04-13T03:51:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-13T03:51:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_base": { 2024-04-13T03:51:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_base", 2024-04-13T03:51:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-13T03:51:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Container image name, without the tag", 2024-04-13T03:51:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_elasticsearch": { 2024-04-13T03:51:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_elasticsearch", 2024-04-13T03:51:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-13T03:51:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "elasticsearch container image", 2024-04-13T03:51:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_grafana": { 2024-04-13T03:51:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_grafana", 2024-04-13T03:51:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-13T03:51:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_haproxy": { 2024-04-13T03:51:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_haproxy", 2024-04-13T03:51:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-13T03:51:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "HAproxy container image", 2024-04-13T03:51:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_agent": { 2024-04-13T03:51:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_agent", 2024-04-13T03:51:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-13T03:51:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger agent container image", 2024-04-13T03:51:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_collector": { 2024-04-13T03:51:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_collector", 2024-04-13T03:51:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-13T03:51:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger collector container image", 2024-04-13T03:51:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_jaeger_query": { 2024-04-13T03:51:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_jaeger_query", 2024-04-13T03:51:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-13T03:51:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Jaeger query container image", 2024-04-13T03:51:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_keepalived": { 2024-04-13T03:51:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_keepalived", 2024-04-13T03:51:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-13T03:51:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Keepalived container image", 2024-04-13T03:51:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_loki": { 2024-04-13T03:51:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_loki", 2024-04-13T03:51:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-13T03:51:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Loki container image", 2024-04-13T03:51:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_node_exporter": { 2024-04-13T03:51:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_node_exporter", 2024-04-13T03:51:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-13T03:51:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_nvmeof": { 2024-04-13T03:51:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_nvmeof", 2024-04-13T03:51:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-13T03:51:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Nvme-of container image", 2024-04-13T03:51:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_prometheus": { 2024-04-13T03:51:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_prometheus", 2024-04-13T03:51:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-13T03:51:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Prometheus container image", 2024-04-13T03:51:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_promtail": { 2024-04-13T03:51:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_promtail", 2024-04-13T03:51:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Promtail container image", 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_samba": { 2024-04-13T03:51:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_samba", 2024-04-13T03:51:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-13T03:51:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Samba/SMB container image", 2024-04-13T03:51:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image_snmp_gateway": { 2024-04-13T03:51:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_image_snmp_gateway", 2024-04-13T03:51:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-13T03:51:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "SNMP Gateway container image", 2024-04-13T03:51:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "container_init": { 2024-04-13T03:51:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "container_init", 2024-04-13T03:51:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Run podman/docker with `--init`", 2024-04-13T03:51:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "daemon_cache_timeout": { 2024-04-13T03:51:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "daemon_cache_timeout", 2024-04-13T03:51:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-13T03:51:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_cephadm_command_timeout": { 2024-04-13T03:51:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_cephadm_command_timeout", 2024-04-13T03:51:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "900", 2024-04-13T03:51:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-13T03:51:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "default_registry": { 2024-04-13T03:51:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default_registry", 2024-04-13T03:51:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "docker.io", 2024-04-13T03:51:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-13T03:51:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "device_cache_timeout": { 2024-04-13T03:51:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_cache_timeout", 2024-04-13T03:51:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1800", 2024-04-13T03:51:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache device inventory", 2024-04-13T03:51:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "device_enhanced_scan": { 2024-04-13T03:51:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_enhanced_scan", 2024-04-13T03:51:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-13T03:51:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "facts_cache_timeout": { 2024-04-13T03:51:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "facts_cache_timeout", 2024-04-13T03:51:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "seconds to cache host facts data", 2024-04-13T03:51:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "grafana_dashboards_path": { 2024-04-13T03:51:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "grafana_dashboards_path", 2024-04-13T03:51:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-13T03:51:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-13T03:51:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "host_check_interval": { 2024-04-13T03:51:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "host_check_interval", 2024-04-13T03:51:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to perform a host check", 2024-04-13T03:51:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "hw_monitoring": { 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hw_monitoring", 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-13T03:51:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "inventory_list_all": { 2024-04-13T03:51:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "inventory_list_all", 2024-04-13T03:51:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-13T03:51:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "log_refresh_metadata": { 2024-04-13T03:51:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_refresh_metadata", 2024-04-13T03:51:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-13T03:51:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-13T03:51:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-13T03:51:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-13T03:51:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-13T03:51:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "*", 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-13T03:51:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "max_count_per_host": { 2024-04-13T03:51:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_count_per_host", 2024-04-13T03:51:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of daemons per service per host", 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd_draining_count": { 2024-04-13T03:51:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_osd_draining_count", 2024-04-13T03:51:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-13T03:51:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "migration_current": { 2024-04-13T03:51:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "migration_current", 2024-04-13T03:51:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "internal - do not modify", 2024-04-13T03:51:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "mode": { 2024-04-13T03:51:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mode", 2024-04-13T03:51:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "root", 2024-04-13T03:51:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm-package", 2024-04-13T03:51:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "root" 2024-04-13T03:51:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "mode for remote execution of cephadm", 2024-04-13T03:51:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "oob_default_addr": { 2024-04-13T03:51:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "oob_default_addr", 2024-04-13T03:51:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "169.254.1.1", 2024-04-13T03:51:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-13T03:51:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "prometheus_alerts_path": { 2024-04-13T03:51:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus_alerts_path", 2024-04-13T03:51:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-13T03:51:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-13T03:51:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_insecure": { 2024-04-13T03:51:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_insecure", 2024-04-13T03:51:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-13T03:51:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_password": { 2024-04-13T03:51:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_password", 2024-04-13T03:51:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-13T03:51:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_url": { 2024-04-13T03:51:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_url", 2024-04-13T03:51:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "registry_username": { 2024-04-13T03:51:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "registry_username", 2024-04-13T03:51:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-13T03:51:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "secure_monitoring_stack": { 2024-04-13T03:51:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "secure_monitoring_stack", 2024-04-13T03:51:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-13T03:51:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "service_discovery_port": { 2024-04-13T03:51:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "service_discovery_port", 2024-04-13T03:51:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8765", 2024-04-13T03:51:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "cephadm service discovery port", 2024-04-13T03:51:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "ssh_config_file": { 2024-04-13T03:51:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssh_config_file", 2024-04-13T03:51:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-13T03:51:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "use_agent": { 2024-04-13T03:51:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_agent", 2024-04-13T03:51:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-13T03:51:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "use_repo_digest": { 2024-04-13T03:51:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "use_repo_digest", 2024-04-13T03:51:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-13T03:51:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_failed_host_check": { 2024-04-13T03:51:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_failed_host_check", 2024-04-13T03:51:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if the host check fails", 2024-04-13T03:51:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_daemons": { 2024-04-13T03:51:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_daemons", 2024-04-13T03:51:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-13T03:51:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_on_stray_hosts": { 2024-04-13T03:51:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_on_stray_hosts", 2024-04-13T03:51:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-13T03:51:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crash", 2024-04-13T03:51:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "retain_interval": { 2024-04-13T03:51:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retain_interval", 2024-04-13T03:51:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "31536000", 2024-04-13T03:51:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-13T03:51:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_recent_interval": { 2024-04-13T03:51:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_recent_interval", 2024-04-13T03:51:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1209600", 2024-04-13T03:51:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-13T03:51:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dashboard", 2024-04-13T03:51:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-13T03:51:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-13T03:51:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_HOST": { 2024-04-13T03:51:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-13T03:51:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-13T03:51:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-13T03:51:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_ENABLED": { 2024-04-13T03:51:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_ENABLED", 2024-04-13T03:51:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-13T03:51:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-13T03:51:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "ENABLE_BROWSABLE_API": { 2024-04-13T03:51:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-13T03:51:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-13T03:51:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-13T03:51:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-13T03:51:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-13T03:51:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-13T03:51:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-13T03:51:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-13T03:51:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-13T03:51:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-13T03:51:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-13T03:51:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-13T03:51:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-13T03:51:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-13T03:51:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-13T03:51:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-13T03:51:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-13T03:51:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_PASSWORD": { 2024-04-13T03:51:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-13T03:51:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-13T03:51:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-13T03:51:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-13T03:51:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_URL": { 2024-04-13T03:51:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_URL", 2024-04-13T03:51:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_API_USERNAME": { 2024-04-13T03:51:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-13T03:51:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-13T03:51:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-13T03:51:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-13T03:51:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-13T03:51:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-13T03:51:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-13T03:51:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-13T03:51:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-13T03:51:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-13T03:51:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-13T03:51:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-13T03:51:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "[]", 2024-04-13T03:51:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "MULTICLUSTER_CONFIG": { 2024-04-13T03:51:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-13T03:51:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "{}", 2024-04-13T03:51:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_HOST": { 2024-04-13T03:51:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-13T03:51:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-13T03:51:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-13T03:51:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-13T03:51:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-13T03:51:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-13T03:51:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-13T03:51:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-13T03:51:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-13T03:51:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-13T03:51:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-13T03:51:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-13T03:51:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-13T03:51:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-13T03:51:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-13T03:51:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-13T03:51:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-13T03:51:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_ENABLED": { 2024-04-13T03:51:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-13T03:51:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-13T03:51:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-13T03:51:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-13T03:51:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-13T03:51:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-13T03:51:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-13T03:51:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-13T03:51:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8", 2024-04-13T03:51:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-13T03:51:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-13T03:51:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "45", 2024-04-13T03:51:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ACCESS_KEY": { 2024-04-13T03:51:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-13T03:51:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-13T03:51:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-13T03:51:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "admin", 2024-04-13T03:51:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SECRET_KEY": { 2024-04-13T03:51:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-13T03:51:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "RGW_API_SSL_VERIFY": { 2024-04-13T03:51:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-13T03:51:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "UNSAFE_TLS_v1_2": { 2024-04-13T03:51:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-13T03:51:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-13T03:51:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-13T03:51:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-13T03:51:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-13T03:51:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "cross_origin_url": { 2024-04-13T03:51:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cross_origin_url", 2024-04-13T03:51:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "crt_file": { 2024-04-13T03:51:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "crt_file", 2024-04-13T03:51:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.471 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.471 INFO:teuthology.orchestra.run.smithi016.stdout: "debug": { 2024-04-13T03:51:17.471 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "debug", 2024-04-13T03:51:17.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.471 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.471 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.472 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.472 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.472 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.472 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Enable/disable debug options", 2024-04-13T03:51:17.472 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.472 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.472 INFO:teuthology.orchestra.run.smithi016.stdout: "jwt_token_ttl": { 2024-04-13T03:51:17.473 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "jwt_token_ttl", 2024-04-13T03:51:17.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.473 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.473 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "28800", 2024-04-13T03:51:17.473 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.473 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.473 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.473 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.474 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.474 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.474 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-13T03:51:17.474 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-13T03:51:17.474 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.474 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.475 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.475 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.475 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.475 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.475 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.475 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.475 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.476 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.476 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.476 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.476 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.476 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.476 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.476 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.476 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.477 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.477 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.477 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.477 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.477 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.477 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.477 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.477 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.478 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.478 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.478 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.478 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.478 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.478 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.478 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.479 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.479 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.479 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.479 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.479 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.479 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.479 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.479 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.480 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.480 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.480 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.480 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.480 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.480 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.481 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.482 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.482 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.482 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.482 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.482 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.482 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.482 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: "motd": { 2024-04-13T03:51:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "motd", 2024-04-13T03:51:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.485 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "The message of the day", 2024-04-13T03:51:17.485 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.485 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.485 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_resolve_ip_addr": { 2024-04-13T03:51:17.485 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "redirect_resolve_ip_addr", 2024-04-13T03:51:17.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.486 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.486 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.486 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.486 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.486 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.486 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.486 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.487 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.487 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-13T03:51:17.487 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-13T03:51:17.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.487 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.487 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-13T03:51:17.488 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.488 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.488 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.488 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.488 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.488 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.488 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.488 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-13T03:51:17.489 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-13T03:51:17.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.489 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.489 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8080", 2024-04-13T03:51:17.489 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.489 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.489 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.489 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.490 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.490 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.490 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-13T03:51:17.490 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-13T03:51:17.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.490 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.491 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.491 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.491 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.491 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.491 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.491 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.491 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.492 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl_server_port": { 2024-04-13T03:51:17.492 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl_server_port", 2024-04-13T03:51:17.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.492 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.492 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.492 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8443", 2024-04-13T03:51:17.492 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.492 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.492 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.493 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.493 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.493 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.493 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.493 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-13T03:51:17.493 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-13T03:51:17.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.493 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.494 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.494 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "redirect", 2024-04-13T03:51:17.494 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.494 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.494 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.494 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.494 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect" 2024-04-13T03:51:17.494 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.494 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.495 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.495 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.495 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.495 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-13T03:51:17.495 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-13T03:51:17.495 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.495 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.495 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-13T03:51:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-13T03:51:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-13T03:51:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: "url_prefix": { 2024-04-13T03:51:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url_prefix", 2024-04-13T03:51:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.497 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.498 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.498 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.498 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.498 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-13T03:51:17.499 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.499 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.499 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.499 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_monitoring": { 2024-04-13T03:51:17.499 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_monitoring", 2024-04-13T03:51:17.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.499 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.499 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.500 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.500 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.500 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.500 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "monitor device health metrics", 2024-04-13T03:51:17.500 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.500 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.500 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.501 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.501 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.501 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.501 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.501 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.501 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.501 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.502 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.502 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.502 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.502 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.502 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.502 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.502 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.502 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.503 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.503 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.503 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.503 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.503 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.503 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.503 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.504 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.504 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.504 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.504 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.504 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.504 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.504 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.504 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.505 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.505 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.505 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.505 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.505 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.505 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.505 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.505 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.505 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.506 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.506 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.506 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.506 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.506 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.506 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.506 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.506 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.506 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.507 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.507 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.507 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.507 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.507 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.507 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.508 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.508 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.508 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.508 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.508 INFO:teuthology.orchestra.run.smithi016.stdout: "mark_out_threshold": { 2024-04-13T03:51:17.508 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mark_out_threshold", 2024-04-13T03:51:17.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.509 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.509 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "2419200", 2024-04-13T03:51:17.509 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.509 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.509 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.509 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-13T03:51:17.509 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.509 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.510 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.510 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": { 2024-04-13T03:51:17.510 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pool_name", 2024-04-13T03:51:17.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.510 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.510 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.510 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "device_health_metrics", 2024-04-13T03:51:17.511 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.511 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.511 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.511 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-13T03:51:17.511 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.511 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.511 INFO:teuthology.orchestra.run.smithi016.stdout: "retention_period": { 2024-04-13T03:51:17.512 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "retention_period", 2024-04-13T03:51:17.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.512 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.512 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15552000", 2024-04-13T03:51:17.512 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.512 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.512 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.512 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long to retain device health metrics", 2024-04-13T03:51:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_frequency": { 2024-04-13T03:51:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_frequency", 2024-04-13T03:51:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.513 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.514 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-13T03:51:17.514 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.514 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.514 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.514 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-13T03:51:17.514 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.514 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.515 INFO:teuthology.orchestra.run.smithi016.stdout: "self_heal": { 2024-04-13T03:51:17.515 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "self_heal", 2024-04-13T03:51:17.515 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.515 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.515 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.515 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.515 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.515 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.516 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.516 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-13T03:51:17.516 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.516 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.516 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:17.516 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:17.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.517 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.517 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:17.517 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.517 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.517 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.517 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how frequently to wake up and check device health", 2024-04-13T03:51:17.517 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.518 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.518 INFO:teuthology.orchestra.run.smithi016.stdout: "warn_threshold": { 2024-04-13T03:51:17.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "warn_threshold", 2024-04-13T03:51:17.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.518 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.518 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.518 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7257600", 2024-04-13T03:51:17.518 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.519 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.519 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.519 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-13T03:51:17.519 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.519 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.519 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "diskprediction_local", 2024-04-13T03:51:17.520 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.520 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.520 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.520 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.520 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.520 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.521 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.521 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.521 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.521 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.521 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.521 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.521 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.521 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.522 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.522 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.522 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.522 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.522 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.522 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.522 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.523 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.523 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.523 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.523 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.523 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.523 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.523 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.524 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.524 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.524 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.524 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.524 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.524 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.524 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.525 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.525 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.525 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.525 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.525 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.525 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.525 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.525 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.526 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.526 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.526 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.526 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.526 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.526 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.527 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.527 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.527 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.527 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.527 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.527 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.527 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.528 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.528 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.528 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.528 INFO:teuthology.orchestra.run.smithi016.stdout: "predict_interval": { 2024-04-13T03:51:17.528 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predict_interval", 2024-04-13T03:51:17.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.528 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.529 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.529 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "86400", 2024-04-13T03:51:17.529 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.529 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.529 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.529 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.529 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.529 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.529 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.530 INFO:teuthology.orchestra.run.smithi016.stdout: "predictor_model": { 2024-04-13T03:51:17.530 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "predictor_model", 2024-04-13T03:51:17.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.530 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.530 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.530 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "prophetstor", 2024-04-13T03:51:17.530 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:17.531 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "600", 2024-04-13T03:51:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.532 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.533 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.533 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "influx", 2024-04-13T03:51:17.533 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": false, 2024-04-13T03:51:17.533 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "influxdb python module not found", 2024-04-13T03:51:17.534 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.534 INFO:teuthology.orchestra.run.smithi016.stdout: "batch_size": { 2024-04-13T03:51:17.534 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "batch_size", 2024-04-13T03:51:17.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.534 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.534 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5000", 2024-04-13T03:51:17.534 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.534 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.535 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.535 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-13T03:51:17.535 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.535 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.535 INFO:teuthology.orchestra.run.smithi016.stdout: "database": { 2024-04-13T03:51:17.535 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "database", 2024-04-13T03:51:17.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.536 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.536 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "ceph", 2024-04-13T03:51:17.536 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.536 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.536 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.536 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-13T03:51:17.536 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.537 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.537 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": { 2024-04-13T03:51:17.537 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "hostname", 2024-04-13T03:51:17.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.537 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.537 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.537 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.537 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.538 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.538 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.538 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server hostname", 2024-04-13T03:51:17.538 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.538 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.538 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:17.539 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:17.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.539 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.539 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.539 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "30", 2024-04-13T03:51:17.539 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "5", 2024-04-13T03:51:17.539 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.539 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.539 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-13T03:51:17.540 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.540 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.540 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.540 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.540 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.540 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.540 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.541 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.541 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.541 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.541 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.541 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.541 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.541 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.541 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.541 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.542 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.542 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.542 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.542 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.542 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.542 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.542 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.543 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.543 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.543 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.543 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.543 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.543 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.543 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.544 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.544 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.544 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.544 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.544 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.545 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.545 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.545 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.545 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.545 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.545 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.545 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.545 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.545 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.546 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.546 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.546 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.546 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.546 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.546 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.547 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.547 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.547 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.547 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.547 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.547 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.547 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.547 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.548 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.548 INFO:teuthology.orchestra.run.smithi016.stdout: "password": { 2024-04-13T03:51:17.548 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "password", 2024-04-13T03:51:17.548 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.548 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.548 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "password of InfluxDB server user", 2024-04-13T03:51:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.549 INFO:teuthology.orchestra.run.smithi016.stdout: "port": { 2024-04-13T03:51:17.550 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "port", 2024-04-13T03:51:17.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.550 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.550 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "8086", 2024-04-13T03:51:17.550 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.550 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.550 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.550 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "InfluxDB server port", 2024-04-13T03:51:17.551 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.551 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.551 INFO:teuthology.orchestra.run.smithi016.stdout: "ssl": { 2024-04-13T03:51:17.551 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ssl", 2024-04-13T03:51:17.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.551 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.552 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "false", 2024-04-13T03:51:17.552 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.552 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.552 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.552 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-13T03:51:17.552 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.552 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.553 INFO:teuthology.orchestra.run.smithi016.stdout: "threads": { 2024-04-13T03:51:17.553 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threads", 2024-04-13T03:51:17.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.553 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.553 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:17.553 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:17.553 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "32", 2024-04-13T03:51:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-13T03:51:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: "username": { 2024-04-13T03:51:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "username", 2024-04-13T03:51:17.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "username of InfluxDB server user", 2024-04-13T03:51:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.555 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.556 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.556 INFO:teuthology.orchestra.run.smithi016.stdout: "verify_ssl": { 2024-04-13T03:51:17.556 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "verify_ssl", 2024-04-13T03:51:17.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.556 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.556 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "true", 2024-04-13T03:51:17.556 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.557 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.557 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.557 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-13T03:51:17.557 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.557 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.558 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "insights", 2024-04-13T03:51:17.558 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.558 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.558 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.558 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.558 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.558 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.558 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.559 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.559 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.559 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.559 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.559 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.559 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.559 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.559 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.560 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.560 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.560 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.560 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.560 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.560 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.560 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.561 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.561 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.561 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.561 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.561 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.561 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.561 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.562 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.562 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.562 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.562 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.562 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.563 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.563 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.563 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.563 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.563 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.563 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.563 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.563 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.563 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.564 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.564 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.564 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.564 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.564 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.564 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.565 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.565 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.565 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.565 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.565 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.565 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.565 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.566 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.566 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.566 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "iostat", 2024-04-13T03:51:17.566 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.567 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.569 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.570 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.570 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.570 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.570 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.570 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.570 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.570 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.570 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.571 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.571 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.571 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.571 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.571 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.571 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.571 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.572 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.572 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.572 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.572 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.572 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.572 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.572 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.572 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.572 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.573 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.574 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.574 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.574 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.574 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.574 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.574 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.574 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "k8sevents", 2024-04-13T03:51:17.575 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.575 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.575 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.575 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_event_retention_days": { 2024-04-13T03:51:17.575 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "ceph_event_retention_days", 2024-04-13T03:51:17.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.576 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.576 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "7", 2024-04-13T03:51:17.576 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.576 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.576 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.576 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-13T03:51:17.576 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.577 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.577 INFO:teuthology.orchestra.run.smithi016.stdout: "config_check_secs": { 2024-04-13T03:51:17.577 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "config_check_secs", 2024-04-13T03:51:17.577 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.577 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.577 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.577 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:17.578 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "10", 2024-04-13T03:51:17.578 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.578 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.578 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-13T03:51:17.578 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.578 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.578 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.578 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.579 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.579 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.579 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.579 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.579 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.579 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.579 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.580 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.581 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.581 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.581 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.581 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.581 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.581 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.582 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.582 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.582 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.582 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.582 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.582 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.582 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.583 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.583 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.583 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.583 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.583 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.583 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.583 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.584 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.584 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.584 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.584 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.584 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.584 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.584 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.584 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.585 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.585 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.585 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.585 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.585 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.585 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.585 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.585 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.586 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.586 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.586 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.586 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.586 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "localpool", 2024-04-13T03:51:17.587 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.587 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.587 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.587 INFO:teuthology.orchestra.run.smithi016.stdout: "failure_domain": { 2024-04-13T03:51:17.587 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "failure_domain", 2024-04-13T03:51:17.587 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.587 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.587 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.588 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "host", 2024-04-13T03:51:17.588 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.588 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.588 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.588 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "failure domain for any created local pool", 2024-04-13T03:51:17.588 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-13T03:51:17.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.588 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.589 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.589 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.589 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.589 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.589 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.589 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.589 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.590 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.590 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.590 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.590 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.590 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.590 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.590 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.590 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.590 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.591 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.591 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.591 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.591 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.591 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.592 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.592 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.592 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.592 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.592 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.592 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.592 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.592 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.593 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.593 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.593 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.593 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.593 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.593 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.594 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.594 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.594 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.594 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.594 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.594 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.594 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.594 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.594 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.595 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.595 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.595 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.595 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.595 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.595 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.596 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.596 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.596 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.596 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.596 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.596 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": { 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "min_size", 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default min_size for any created local pool", 2024-04-13T03:51:17.597 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-13T03:51:17.598 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.598 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.598 INFO:teuthology.orchestra.run.smithi016.stdout: "num_rep": { 2024-04-13T03:51:17.598 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "num_rep", 2024-04-13T03:51:17.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.598 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.599 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3", 2024-04-13T03:51:17.599 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.599 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.599 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.599 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default replica count for any created local pool", 2024-04-13T03:51:17.599 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.599 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.600 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": { 2024-04-13T03:51:17.600 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_num", 2024-04-13T03:51:17.600 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.600 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.600 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "128", 2024-04-13T03:51:17.600 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.600 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.600 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.601 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "default pg_num for any created local pool", 2024-04-13T03:51:17.601 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.601 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.601 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.601 INFO:teuthology.orchestra.run.smithi016.stdout: "prefix": { 2024-04-13T03:51:17.601 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prefix", 2024-04-13T03:51:17.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.601 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.602 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.602 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.602 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.602 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.602 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "name prefix for any created local pool", 2024-04-13T03:51:17.602 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.602 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.602 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.603 INFO:teuthology.orchestra.run.smithi016.stdout: "subtree": { 2024-04-13T03:51:17.603 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "subtree", 2024-04-13T03:51:17.603 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.603 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.603 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.603 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "rack", 2024-04-13T03:51:17.603 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.603 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.604 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.604 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-13T03:51:17.604 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-13T03:51:17.604 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.604 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.605 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mds_autoscaler", 2024-04-13T03:51:17.605 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.605 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.605 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.605 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.605 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.605 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.606 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.606 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.606 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.606 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.606 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.606 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.606 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.606 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.606 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.607 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.607 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.607 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.607 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.607 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.607 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.607 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.608 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.608 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.608 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.608 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.608 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.608 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.608 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.608 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.608 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.609 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.609 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.609 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.609 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.609 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.609 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.610 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.610 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.610 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.610 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.610 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.610 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.610 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.610 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.610 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.611 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.611 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.611 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.611 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.611 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.611 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.612 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.612 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.612 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.612 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.612 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.612 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.612 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.613 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.613 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirroring", 2024-04-13T03:51:17.613 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.613 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.613 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.614 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.615 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.615 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.615 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.615 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.615 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.615 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.615 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.615 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.615 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.616 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.617 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.617 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.617 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.617 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.617 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.617 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.618 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.618 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.618 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.618 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.618 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.618 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.618 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.618 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.619 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.619 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.619 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.619 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.619 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.619 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.619 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.619 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.619 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.620 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.620 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.620 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.620 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.620 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.620 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.622 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "nfs", 2024-04-13T03:51:17.622 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.622 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.622 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.622 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.622 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.623 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.623 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.623 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.623 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.623 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.623 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.624 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.624 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.624 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.624 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.624 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.624 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.624 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.624 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.625 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.625 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.625 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.625 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.625 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.625 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.625 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.626 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.627 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.627 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.627 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.627 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.627 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.627 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.627 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.628 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.628 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.628 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.628 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.628 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.628 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.628 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.628 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.629 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.629 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.629 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.629 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.629 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.629 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.630 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.630 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.630 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.630 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.630 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.630 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: "fail_fs": { 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "fail_fs", 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-13T03:51:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.633 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.634 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.635 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.636 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.637 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.637 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.637 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.637 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.637 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.637 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.637 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.637 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.638 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.638 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.638 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.638 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.638 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.638 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.638 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.638 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.639 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.639 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.639 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.639 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.639 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator": { 2024-04-13T03:51:17.640 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "orchestrator", 2024-04-13T03:51:17.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.641 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.641 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.641 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.641 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.641 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.641 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm", 2024-04-13T03:51:17.641 INFO:teuthology.orchestra.run.smithi016.stdout: "rook", 2024-04-13T03:51:17.642 INFO:teuthology.orchestra.run.smithi016.stdout: "test_orchestrator" 2024-04-13T03:51:17.642 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.642 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Orchestrator backend", 2024-04-13T03:51:17.642 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.642 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_perf_query", 2024-04-13T03:51:17.643 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.643 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.643 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.643 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.643 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.643 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.644 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.644 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.644 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.644 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.644 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.644 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.644 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.644 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.645 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.646 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.646 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.646 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.646 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.646 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.646 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.647 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.647 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.647 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.647 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.647 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.647 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.648 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.649 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.649 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.649 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.649 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.649 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.649 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.649 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.650 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.651 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "osd_support", 2024-04-13T03:51:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.652 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.653 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.653 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.653 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.653 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.653 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.653 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.653 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.653 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.653 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.654 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.654 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.654 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.654 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.654 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.654 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.654 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.655 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.655 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.655 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.655 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.655 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.655 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.655 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.656 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.657 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.657 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.657 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.657 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.657 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.657 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.657 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.657 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.657 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.658 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.658 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.658 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.658 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.658 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.658 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.660 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "pg_autoscaler", 2024-04-13T03:51:17.660 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.660 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.660 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.660 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.660 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.661 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.661 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.661 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.661 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.661 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.661 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.661 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.661 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.662 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.662 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.662 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.662 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.662 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.662 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.662 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.662 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.662 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.663 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.664 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.664 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.664 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.664 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.664 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.664 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.665 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.665 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.665 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.665 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.665 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.665 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.665 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.665 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.665 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.666 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.666 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.666 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.666 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.666 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.666 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.666 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.667 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.667 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.667 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.667 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.667 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.667 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.667 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.670 INFO:teuthology.orchestra.run.smithi016.stdout: "threshold": { 2024-04-13T03:51:17.670 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "threshold", 2024-04-13T03:51:17.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:17.670 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.670 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "3.0", 2024-04-13T03:51:17.670 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1.0", 2024-04-13T03:51:17.670 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.670 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.671 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "scaling threshold", 2024-04-13T03:51:17.671 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-13T03:51:17.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.671 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "progress", 2024-04-13T03:51:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_pg_recovery_event": { 2024-04-13T03:51:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_pg_recovery_event", 2024-04-13T03:51:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.673 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.673 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.673 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.673 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.673 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-13T03:51:17.673 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.673 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.673 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.674 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-13T03:51:17.674 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-13T03:51:17.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.674 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.674 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.674 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.674 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.674 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.675 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.675 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.675 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.675 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.675 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.675 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.676 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.676 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.676 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.676 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.676 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.676 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.676 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.676 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.677 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.677 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.677 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.677 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.677 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.677 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.677 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.678 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.678 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.678 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.678 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.678 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.678 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.678 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.678 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.679 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.679 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.679 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.679 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.679 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.680 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.680 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.680 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.680 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.680 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.680 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.680 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.680 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.680 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.681 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.681 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.681 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.681 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.681 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.681 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.681 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.681 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.682 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.682 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.682 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.682 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.682 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.682 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.682 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.683 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.683 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.683 INFO:teuthology.orchestra.run.smithi016.stdout: "max_completed_events": { 2024-04-13T03:51:17.683 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_completed_events", 2024-04-13T03:51:17.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.683 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.683 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "50", 2024-04-13T03:51:17.684 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.684 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.684 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.684 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "number of past completed events to remember", 2024-04-13T03:51:17.684 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.684 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.684 INFO:teuthology.orchestra.run.smithi016.stdout: "sleep_interval": { 2024-04-13T03:51:17.685 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "sleep_interval", 2024-04-13T03:51:17.685 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.685 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.685 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.685 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "5", 2024-04-13T03:51:17.685 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.685 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.685 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.685 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "how long the module is going to sleep", 2024-04-13T03:51:17.686 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.686 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.686 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "prometheus", 2024-04-13T03:51:17.686 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.687 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.687 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.687 INFO:teuthology.orchestra.run.smithi016.stdout: "cache": { 2024-04-13T03:51:17.687 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "cache", 2024-04-13T03:51:17.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.687 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.687 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.687 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.688 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.688 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.688 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.688 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.688 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.688 INFO:teuthology.orchestra.run.smithi016.stdout: "exclude_perf_counters": { 2024-04-13T03:51:17.688 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "exclude_perf_counters", 2024-04-13T03:51:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-13T03:51:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-13T03:51:17.690 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.690 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.690 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.690 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.690 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.690 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.691 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.691 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.691 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.691 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.691 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.691 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.691 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.691 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.691 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.692 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.692 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.692 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.692 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.692 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.692 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.693 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.694 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.694 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.694 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.694 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.694 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.694 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.694 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.695 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.695 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.695 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.695 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.695 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.695 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.695 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.695 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.696 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.696 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.696 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.696 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.696 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.696 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.696 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.697 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.697 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.697 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.697 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.697 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.697 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.697 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools": { 2024-04-13T03:51:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools", 2024-04-13T03:51:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-13T03:51:17.699 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-13T03:51:17.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.700 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.700 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "300", 2024-04-13T03:51:17.700 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.700 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.700 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.700 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.700 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.701 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.701 INFO:teuthology.orchestra.run.smithi016.stdout: "scrape_interval": { 2024-04-13T03:51:17.701 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "scrape_interval", 2024-04-13T03:51:17.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:17.701 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.701 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15.0", 2024-04-13T03:51:17.702 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.702 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.702 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.702 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.702 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.702 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.702 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.702 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-13T03:51:17.703 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-13T03:51:17.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.703 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.703 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "::", 2024-04-13T03:51:17.703 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.703 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.703 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.703 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-13T03:51:17.704 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.704 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.704 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.704 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-13T03:51:17.704 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-13T03:51:17.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.704 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.705 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "9283", 2024-04-13T03:51:17.705 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.705 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.705 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.705 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-13T03:51:17.705 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.705 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.706 INFO:teuthology.orchestra.run.smithi016.stdout: "stale_cache_strategy": { 2024-04-13T03:51:17.706 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stale_cache_strategy", 2024-04-13T03:51:17.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.706 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.706 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "log", 2024-04-13T03:51:17.706 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.706 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.706 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.707 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.707 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.707 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.707 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_behaviour": { 2024-04-13T03:51:17.707 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_behaviour", 2024-04-13T03:51:17.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.707 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.708 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.708 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "default", 2024-04-13T03:51:17.708 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.708 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.708 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.708 INFO:teuthology.orchestra.run.smithi016.stdout: "default", 2024-04-13T03:51:17.708 INFO:teuthology.orchestra.run.smithi016.stdout: "error" 2024-04-13T03:51:17.708 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.708 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.709 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.709 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.709 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.709 INFO:teuthology.orchestra.run.smithi016.stdout: "standby_error_status_code": { 2024-04-13T03:51:17.709 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "standby_error_status_code", 2024-04-13T03:51:17.709 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.709 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.709 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.710 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "500", 2024-04-13T03:51:17.710 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "400", 2024-04-13T03:51:17.710 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "599", 2024-04-13T03:51:17.710 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.710 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.710 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.710 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.710 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.711 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-13T03:51:17.711 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.711 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.711 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.711 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.711 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.712 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.712 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.712 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.712 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.712 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.712 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.712 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.714 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.714 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.714 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.714 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.714 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.714 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.714 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.714 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.714 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.715 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.715 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.715 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.715 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.715 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.715 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.716 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.716 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.716 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.716 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.716 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.716 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.716 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.716 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.717 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.717 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.717 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.717 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.717 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.717 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.717 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.718 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.718 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.718 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.718 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.718 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.718 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.718 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.718 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.719 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.719 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.719 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.719 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_snap_create": { 2024-04-13T03:51:17.719 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_snap_create", 2024-04-13T03:51:17.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.719 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.720 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10", 2024-04-13T03:51:17.720 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.720 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.720 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.720 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.720 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.720 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.721 INFO:teuthology.orchestra.run.smithi016.stdout: "mirror_snapshot_schedule": { 2024-04-13T03:51:17.721 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mirror_snapshot_schedule", 2024-04-13T03:51:17.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.721 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.721 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.721 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.721 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.721 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.722 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.722 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.722 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.722 INFO:teuthology.orchestra.run.smithi016.stdout: "trash_purge_schedule": { 2024-04-13T03:51:17.722 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "trash_purge_schedule", 2024-04-13T03:51:17.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.722 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.723 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.723 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.723 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.723 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.723 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.723 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.723 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.723 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.724 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "restful", 2024-04-13T03:51:17.724 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.724 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.724 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.724 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_auth": { 2024-04-13T03:51:17.725 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enable_auth", 2024-04-13T03:51:17.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.725 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.725 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.725 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.725 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.725 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.725 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.726 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.726 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.726 INFO:teuthology.orchestra.run.smithi016.stdout: "key_file": { 2024-04-13T03:51:17.726 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "key_file", 2024-04-13T03:51:17.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.726 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.727 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.727 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.727 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.727 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.727 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.727 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.727 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.727 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.728 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.728 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.728 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.728 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.728 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.728 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.729 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.729 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.729 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.729 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.729 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.729 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.729 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.729 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.729 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.730 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.730 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.730 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.730 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.730 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.731 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.732 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.732 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.732 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.732 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.732 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.732 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.732 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.732 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.733 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.733 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.733 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.733 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.733 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.733 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.733 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.733 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.733 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.734 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.734 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.734 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.734 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.734 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.734 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.735 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.735 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.735 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.735 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.735 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.735 INFO:teuthology.orchestra.run.smithi016.stdout: "server_addr": { 2024-04-13T03:51:17.735 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_addr", 2024-04-13T03:51:17.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.736 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.736 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.736 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.736 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.736 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.736 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.736 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.737 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.737 INFO:teuthology.orchestra.run.smithi016.stdout: "server_port": { 2024-04-13T03:51:17.737 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "server_port", 2024-04-13T03:51:17.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.737 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.737 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.737 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.738 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.738 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.738 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.738 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.738 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.738 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.738 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.739 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rgw", 2024-04-13T03:51:17.739 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.739 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.739 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.739 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.739 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.740 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.740 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.740 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.740 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.740 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.740 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.742 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.742 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.742 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.742 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.742 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.742 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.743 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.743 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.743 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.743 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.743 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.743 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.744 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.744 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.744 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.744 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.744 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.744 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.744 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.744 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.744 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.745 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.746 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.747 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.747 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.747 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.747 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.747 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.747 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rook", 2024-04-13T03:51:17.748 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.748 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.748 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.748 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.748 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.748 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.748 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.749 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.749 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.749 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.749 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.749 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.749 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.749 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.749 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.750 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.750 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.750 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.750 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.750 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.750 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.751 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.751 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.751 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.751 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.751 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.751 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.751 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.752 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.752 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.752 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.752 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.752 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.752 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.753 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.754 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.754 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.754 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.754 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.754 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.754 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.754 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.754 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.756 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.756 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_class": { 2024-04-13T03:51:17.756 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "storage_class", 2024-04-13T03:51:17.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.756 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.756 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "local", 2024-04-13T03:51:17.757 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.757 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.757 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.757 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-13T03:51:17.757 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.757 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.758 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "selftest", 2024-04-13T03:51:17.758 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.758 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.758 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.758 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.758 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.759 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.759 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.759 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.759 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.759 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.759 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.760 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.760 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.760 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.760 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.760 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.760 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.760 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.760 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.761 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.762 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.762 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.762 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.762 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.762 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.762 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.762 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.763 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.763 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.763 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.763 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.763 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.763 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.763 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.763 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.764 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.764 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.764 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.764 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.764 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.764 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.764 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.765 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.765 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.765 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.765 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.765 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.765 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.766 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.766 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.766 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.766 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.766 INFO:teuthology.orchestra.run.smithi016.stdout: "roption1": { 2024-04-13T03:51:17.766 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption1", 2024-04-13T03:51:17.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.767 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.767 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.767 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.767 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.767 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.767 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.767 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "roption2": { 2024-04-13T03:51:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "roption2", 2024-04-13T03:51:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "xyz", 2024-04-13T03:51:17.768 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.769 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption1": { 2024-04-13T03:51:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption1", 2024-04-13T03:51:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.770 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption2": { 2024-04-13T03:51:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption2", 2024-04-13T03:51:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption3": { 2024-04-13T03:51:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption3", 2024-04-13T03:51:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "float", 2024-04-13T03:51:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.773 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption4": { 2024-04-13T03:51:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption4", 2024-04-13T03:51:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.774 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption5": { 2024-04-13T03:51:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption5", 2024-04-13T03:51:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.776 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.777 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption6": { 2024-04-13T03:51:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption6", 2024-04-13T03:51:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.778 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "rwoption7": { 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rwoption7", 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "1", 2024-04-13T03:51:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "42", 2024-04-13T03:51:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "testkey": { 2024-04-13T03:51:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testkey", 2024-04-13T03:51:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.781 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "testlkey": { 2024-04-13T03:51:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testlkey", 2024-04-13T03:51:17.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.783 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "testnewline": { 2024-04-13T03:51:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "testnewline", 2024-04-13T03:51:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.784 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-13T03:51:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "allow_m_granularity": { 2024-04-13T03:51:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "allow_m_granularity", 2024-04-13T03:51:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "allow minute scheduled snapshots", 2024-04-13T03:51:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.787 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "dump_on_update": { 2024-04-13T03:51:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "dump_on_update", 2024-04-13T03:51:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.788 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "dump database to debug log on update", 2024-04-13T03:51:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.789 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.790 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.792 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.793 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.794 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.795 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.796 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "stats", 2024-04-13T03:51:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.798 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.799 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.800 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.801 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.802 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.803 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.804 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.805 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "status", 2024-04-13T03:51:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.806 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.807 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.808 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.809 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.810 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.812 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telegraf", 2024-04-13T03:51:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "address": { 2024-04-13T03:51:17.815 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "address", 2024-04-13T03:51:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-13T03:51:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.816 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.817 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "15", 2024-04-13T03:51:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.819 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.821 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.822 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.823 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.825 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "telemetry", 2024-04-13T03:51:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_basic": { 2024-04-13T03:51:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_basic", 2024-04-13T03:51:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.827 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-13T03:51:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_crash": { 2024-04-13T03:51:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_crash", 2024-04-13T03:51:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.829 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-13T03:51:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_device": { 2024-04-13T03:51:17.830 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_device", 2024-04-13T03:51:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-13T03:51:17.831 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_ident": { 2024-04-13T03:51:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_ident", 2024-04-13T03:51:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-13T03:51:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "channel_perf": { 2024-04-13T03:51:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "channel_perf", 2024-04-13T03:51:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.834 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-13T03:51:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "contact": { 2024-04-13T03:51:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "contact", 2024-04-13T03:51:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.835 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.836 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "description": { 2024-04-13T03:51:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "description", 2024-04-13T03:51:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.837 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "device_url": { 2024-04-13T03:51:17.838 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "device_url", 2024-04-13T03:51:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-13T03:51:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.839 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.840 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": { 2024-04-13T03:51:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "enabled", 2024-04-13T03:51:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.842 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.843 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "24", 2024-04-13T03:51:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "8", 2024-04-13T03:51:17.844 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "last_opt_revision": { 2024-04-13T03:51:17.845 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "last_opt_revision", 2024-04-13T03:51:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "1", 2024-04-13T03:51:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.846 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard": { 2024-04-13T03:51:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard", 2024-04-13T03:51:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.847 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.848 INFO:teuthology.orchestra.run.smithi016.stdout: "leaderboard_description": { 2024-04-13T03:51:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "leaderboard_description", 2024-04-13T03:51:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.849 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.850 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.851 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.852 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.853 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.855 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.856 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.857 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "organization": { 2024-04-13T03:51:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "organization", 2024-04-13T03:51:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.858 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.859 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "proxy": { 2024-04-13T03:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "proxy", 2024-04-13T03:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.860 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "url": { 2024-04-13T03:51:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "url", 2024-04-13T03:51:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-13T03:51:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.862 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "test_orchestrator", 2024-04-13T03:51:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.864 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.865 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.866 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.867 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.867 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.867 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.867 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.867 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.867 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.867 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.868 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.868 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.868 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.868 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.868 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.868 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.869 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.869 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.869 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.869 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.869 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.869 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.869 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.869 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.869 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.870 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.870 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.870 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.870 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.870 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.870 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.870 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.871 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.871 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.871 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.871 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.871 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.871 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.871 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.871 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.872 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.872 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.872 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.872 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-13T03:51:17.872 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.872 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.872 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.873 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.873 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.873 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.873 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.873 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.873 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.873 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.873 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.874 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.874 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.874 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.874 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.874 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.874 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.874 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.874 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.874 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.875 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.875 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.875 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.875 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.875 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.876 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.876 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.876 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.876 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.876 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.876 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.876 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.876 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.877 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.877 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.877 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.877 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.877 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.877 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.877 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.878 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.878 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.878 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.878 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.878 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.878 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.878 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.878 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.879 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.880 INFO:teuthology.orchestra.run.smithi016.stdout: "max_concurrent_clones": { 2024-04-13T03:51:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "max_concurrent_clones", 2024-04-13T03:51:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "4", 2024-04-13T03:51:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.881 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-13T03:51:17.882 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.882 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.882 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.882 INFO:teuthology.orchestra.run.smithi016.stdout: "periodic_async_work": { 2024-04-13T03:51:17.882 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "periodic_async_work", 2024-04-13T03:51:17.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.882 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.883 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.883 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.883 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.883 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.883 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Periodically check for async work", 2024-04-13T03:51:17.883 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.883 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.884 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_delay": { 2024-04-13T03:51:17.884 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_delay", 2024-04-13T03:51:17.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.884 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.884 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.884 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "0", 2024-04-13T03:51:17.884 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.884 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.884 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.885 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-13T03:51:17.885 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.885 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.885 INFO:teuthology.orchestra.run.smithi016.stdout: "snapshot_clone_no_wait": { 2024-04-13T03:51:17.885 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snapshot_clone_no_wait", 2024-04-13T03:51:17.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.885 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.886 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "True", 2024-04-13T03:51:17.886 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.886 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.886 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.886 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-13T03:51:17.886 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.887 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.887 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix", 2024-04-13T03:51:17.887 INFO:teuthology.orchestra.run.smithi016.stdout: "can_run": true, 2024-04-13T03:51:17.887 INFO:teuthology.orchestra.run.smithi016.stdout: "error_string": "", 2024-04-13T03:51:17.887 INFO:teuthology.orchestra.run.smithi016.stdout: "module_options": { 2024-04-13T03:51:17.888 INFO:teuthology.orchestra.run.smithi016.stdout: "discovery_interval": { 2024-04-13T03:51:17.888 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "discovery_interval", 2024-04-13T03:51:17.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "uint", 2024-04-13T03:51:17.888 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.888 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "100", 2024-04-13T03:51:17.888 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.888 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.888 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.889 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.889 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.889 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.889 INFO:teuthology.orchestra.run.smithi016.stdout: "identifier": { 2024-04-13T03:51:17.889 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "identifier", 2024-04-13T03:51:17.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.889 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.890 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.890 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.890 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.890 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.890 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.890 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: "interval": { 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "interval", 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "secs", 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "60", 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.891 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.892 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.892 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.892 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.892 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.892 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.892 INFO:teuthology.orchestra.run.smithi016.stdout: "log_level": { 2024-04-13T03:51:17.892 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_level", 2024-04-13T03:51:17.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.893 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.893 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.893 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.893 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.893 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.893 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.893 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.894 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.894 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.894 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.894 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.894 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.894 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.894 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.894 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.895 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster": { 2024-04-13T03:51:17.895 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster", 2024-04-13T03:51:17.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.895 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.895 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.895 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.896 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.896 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.896 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.896 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.896 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_cluster_level": { 2024-04-13T03:51:17.896 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_cluster_level", 2024-04-13T03:51:17.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "info", 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [ 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "", 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "critical", 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "debug", 2024-04-13T03:51:17.897 INFO:teuthology.orchestra.run.smithi016.stdout: "error", 2024-04-13T03:51:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "info", 2024-04-13T03:51:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "warning" 2024-04-13T03:51:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.898 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": { 2024-04-13T03:51:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "log_to_file", 2024-04-13T03:51:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "bool", 2024-04-13T03:51:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2024-04-13T03:51:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "False", 2024-04-13T03:51:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.899 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_host": { 2024-04-13T03:51:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_host", 2024-04-13T03:51:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.901 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "", 2024-04-13T03:51:17.901 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.901 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.901 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.901 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.901 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.901 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.902 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_port": { 2024-04-13T03:51:17.902 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_port", 2024-04-13T03:51:17.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "int", 2024-04-13T03:51:17.902 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.902 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "10051", 2024-04-13T03:51:17.902 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.902 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.902 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.903 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.903 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.903 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.903 INFO:teuthology.orchestra.run.smithi016.stdout: "zabbix_sender": { 2024-04-13T03:51:17.903 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "zabbix_sender", 2024-04-13T03:51:17.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "str", 2024-04-13T03:51:17.903 INFO:teuthology.orchestra.run.smithi016.stdout: "level": "advanced", 2024-04-13T03:51:17.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 0, 2024-04-13T03:51:17.904 INFO:teuthology.orchestra.run.smithi016.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-13T03:51:17.904 INFO:teuthology.orchestra.run.smithi016.stdout: "min": "", 2024-04-13T03:51:17.904 INFO:teuthology.orchestra.run.smithi016.stdout: "max": "", 2024-04-13T03:51:17.904 INFO:teuthology.orchestra.run.smithi016.stdout: "enum_allowed": [], 2024-04-13T03:51:17.904 INFO:teuthology.orchestra.run.smithi016.stdout: "desc": "", 2024-04-13T03:51:17.904 INFO:teuthology.orchestra.run.smithi016.stdout: "long_desc": "", 2024-04-13T03:51:17.904 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-13T03:51:17.904 INFO:teuthology.orchestra.run.smithi016.stdout: "see_also": [] 2024-04-13T03:51:17.905 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.905 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.905 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.905 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.905 INFO:teuthology.orchestra.run.smithi016.stdout: "services": { 2024-04-13T03:51:17.905 INFO:teuthology.orchestra.run.smithi016.stdout: "dashboard": "https://172.21.15.16:8443/" 2024-04-13T03:51:17.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.905 INFO:teuthology.orchestra.run.smithi016.stdout: "always_on_modules": { 2024-04-13T03:51:17.905 INFO:teuthology.orchestra.run.smithi016.stdout: "octopus": [ 2024-04-13T03:51:17.906 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-13T03:51:17.906 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-13T03:51:17.906 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-13T03:51:17.906 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-13T03:51:17.906 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-13T03:51:17.906 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-13T03:51:17.906 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-13T03:51:17.906 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-13T03:51:17.906 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-13T03:51:17.907 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-13T03:51:17.907 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.907 INFO:teuthology.orchestra.run.smithi016.stdout: "pacific": [ 2024-04-13T03:51:17.907 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-13T03:51:17.907 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-13T03:51:17.907 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-13T03:51:17.907 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-13T03:51:17.907 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-13T03:51:17.907 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-13T03:51:17.908 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-13T03:51:17.908 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-13T03:51:17.908 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-13T03:51:17.908 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-13T03:51:17.908 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.908 INFO:teuthology.orchestra.run.smithi016.stdout: "quincy": [ 2024-04-13T03:51:17.908 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-13T03:51:17.908 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-13T03:51:17.908 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-13T03:51:17.909 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-13T03:51:17.909 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-13T03:51:17.909 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-13T03:51:17.909 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-13T03:51:17.909 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-13T03:51:17.909 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-13T03:51:17.909 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-13T03:51:17.909 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.909 INFO:teuthology.orchestra.run.smithi016.stdout: "reef": [ 2024-04-13T03:51:17.910 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-13T03:51:17.910 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-13T03:51:17.910 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-13T03:51:17.910 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-13T03:51:17.910 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-13T03:51:17.910 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-13T03:51:17.910 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-13T03:51:17.910 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-13T03:51:17.910 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-13T03:51:17.911 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-13T03:51:17.911 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-13T03:51:17.911 INFO:teuthology.orchestra.run.smithi016.stdout: "squid": [ 2024-04-13T03:51:17.911 INFO:teuthology.orchestra.run.smithi016.stdout: "balancer", 2024-04-13T03:51:17.911 INFO:teuthology.orchestra.run.smithi016.stdout: "crash", 2024-04-13T03:51:17.911 INFO:teuthology.orchestra.run.smithi016.stdout: "devicehealth", 2024-04-13T03:51:17.911 INFO:teuthology.orchestra.run.smithi016.stdout: "orchestrator", 2024-04-13T03:51:17.911 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscaler", 2024-04-13T03:51:17.911 INFO:teuthology.orchestra.run.smithi016.stdout: "progress", 2024-04-13T03:51:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_support", 2024-04-13T03:51:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: "status", 2024-04-13T03:51:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: "telemetry", 2024-04-13T03:51:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: "volumes" 2024-04-13T03:51:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: "last_failure_osd_epoch": 95, 2024-04-13T03:51:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: "active_clients": [ 2024-04-13T03:51:17.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "devicehealth", 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3549550744 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.914 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "libcephsqlite", 2024-04-13T03:51:17.914 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-13T03:51:17.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-13T03:51:17.914 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-13T03:51:17.914 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3103335078 2024-04-13T03:51:17.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.914 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:17.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.915 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "rbd_support", 2024-04-13T03:51:17.915 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-13T03:51:17.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-13T03:51:17.915 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-13T03:51:17.915 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 1987331281 2024-04-13T03:51:17.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.916 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:17.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.916 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-13T03:51:17.916 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-13T03:51:17.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-13T03:51:17.916 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-13T03:51:17.916 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2078192262 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "snap_schedule", 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-13T03:51:17.917 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3190613175 2024-04-13T03:51:17.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.918 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:17.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:17.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.918 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "volumes", 2024-04-13T03:51:17.918 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2024-04-13T03:51:17.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-13T03:51:17.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2024-04-13T03:51:17.919 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2024-04-13T03:51:17.919 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2755082335 2024-04-13T03:51:17.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.919 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:17.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:17.919 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-13T03:51:17.919 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:51:17.925 DEBUG:tasks.check_counter:Getting stats from a 2024-04-13T03:51:17.926 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-13T03:51:17.926 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:15 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3960346700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:51:17.926 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:15 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2235974691' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-13T03:51:17.926 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:16 smithi016 ceph-mon[31772]: pgmap v520: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.0 MiB/s wr, 490 op/s 2024-04-13T03:51:17.928 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:15 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3960346700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:51:17.928 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:15 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2235974691' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-13T03:51:17.928 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:16 smithi119 ceph-mon[35835]: pgmap v520: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.0 MiB/s wr, 490 op/s 2024-04-13T03:51:17.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:15 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3960346700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:51:17.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:15 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2235974691' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-13T03:51:17.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:16 smithi188 ceph-mon[36398]: pgmap v520: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.0 MiB/s wr, 490 op/s 2024-04-13T03:51:18.401 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:51:19.133 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:18 smithi016 ceph-mon[31772]: pgmap v521: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 181 op/s 2024-04-13T03:51:19.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:18 smithi119 ceph-mon[35835]: pgmap v521: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 181 op/s 2024-04-13T03:51:19.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:18 smithi188 ceph-mon[36398]: pgmap v521: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 181 op/s 2024-04-13T03:51:20.959 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:51:20.959 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:20.959 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 4, 2024-04-13T03:51:20.959 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 4, 2024-04-13T03:51:20.959 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3024, 2024-04-13T03:51:20.960 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 448, 2024-04-13T03:51:20.960 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-13T03:51:20.960 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-13T03:51:20.960 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.004928087, 2024-04-13T03:51:20.960 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000058061, 2024-04-13T03:51:20.960 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.001437157, 2024-04-13T03:51:20.960 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-13T03:51:20.960 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:20.960 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3, 2024-04-13T03:51:20.961 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.001640019, 2024-04-13T03:51:20.961 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000546673 2024-04-13T03:51:20.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.961 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:20.961 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:20.961 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:20.961 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:20.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.961 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-13T03:51:20.962 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 448 2024-04-13T03:51:20.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.962 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:20.962 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 3, 2024-04-13T03:51:20.962 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 243, 2024-04-13T03:51:20.962 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 480, 2024-04-13T03:51:20.962 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 277344, 2024-04-13T03:51:20.962 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 6, 2024-04-13T03:51:20.962 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.050721020, 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.037397313, 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.006062143, 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 240, 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.040529595, 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000168873 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.963 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:20.964 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:20.964 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:20.964 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:20.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.964 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-13T03:51:20.964 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 277344 2024-04-13T03:51:20.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.964 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:20.964 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 329, 2024-04-13T03:51:20.965 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 318, 2024-04-13T03:51:20.965 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1620304, 2024-04-13T03:51:20.965 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 196400, 2024-04-13T03:51:20.965 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-13T03:51:20.965 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-13T03:51:20.966 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.135786268, 2024-04-13T03:51:20.966 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.052427975, 2024-04-13T03:51:20.966 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.083611990, 2024-04-13T03:51:20.966 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.015503907, 2024-04-13T03:51:20.966 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:20.966 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 317, 2024-04-13T03:51:20.966 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.035650550, 2024-04-13T03:51:20.966 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000112462 2024-04-13T03:51:20.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1620304, 2024-04-13T03:51:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 196400 2024-04-13T03:51:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.967 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-13T03:51:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-13T03:51:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-13T03:51:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-13T03:51:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:20.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:20.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:20.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:20.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:20.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:20.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:20.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:20.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:20.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:20.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:20.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-13T03:51:20.972 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:20.972 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:20.972 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:20.972 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:20.972 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 147930, 2024-04-13T03:51:20.972 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 28, 2024-04-13T03:51:20.972 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-13T03:51:20.972 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-13T03:51:20.972 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-13T03:51:20.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-13T03:51:20.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:20.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:20.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:20.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:20.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-13T03:51:20.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-13T03:51:20.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:20.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:20.974 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:20.974 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-13T03:51:20.974 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-13T03:51:20.974 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-13T03:51:20.974 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:20.974 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-13T03:51:20.974 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:20.974 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-13T03:51:20.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:20.975 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:20.975 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:20.975 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-13T03:51:20.975 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:20.975 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 332, 2024-04-13T03:51:20.975 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1573884, 2024-04-13T03:51:20.975 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:20.975 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 332, 2024-04-13T03:51:20.975 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-13T03:51:20.976 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-13T03:51:20.976 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 332, 2024-04-13T03:51:20.976 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1573884, 2024-04-13T03:51:20.976 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:20.976 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:20.976 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:20.976 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:20.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:20.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:20.977 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:51:21.105 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:20 smithi016 ceph-mon[31772]: pgmap v522: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 155 op/s 2024-04-13T03:51:21.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:20 smithi119 ceph-mon[35835]: pgmap v522: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 155 op/s 2024-04-13T03:51:21.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:20 smithi188 ceph-mon[36398]: pgmap v522: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.0 MiB/s wr, 155 op/s 2024-04-13T03:51:21.871 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-13T03:51:21.871 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-13T03:51:21.872 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-13T03:51:21.872 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-13T03:51:21.872 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-13T03:51:21.872 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-13T03:51:21.872 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-13T03:51:21.872 DEBUG:tasks.check_counter:Getting stats from d 2024-04-13T03:51:21.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-13T03:51:22.376 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:51:23.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:22 smithi016 ceph-mon[31772]: pgmap v523: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-04-13T03:51:23.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:22 smithi119 ceph-mon[35835]: pgmap v523: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-04-13T03:51:23.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:22 smithi188 ceph-mon[36398]: pgmap v523: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-04-13T03:51:24.857 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:51:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2969, 2024-04-13T03:51:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2971, 2024-04-13T03:51:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 52804893, 2024-04-13T03:51:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 944149, 2024-04-13T03:51:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 8, 2024-04-13T03:51:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 4, 2024-04-13T03:51:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.755820222, 2024-04-13T03:51:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.196393214, 2024-04-13T03:51:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.328613412, 2024-04-13T03:51:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.175746847, 2024-04-13T03:51:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2969, 2024-04-13T03:51:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.280336711, 2024-04-13T03:51:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000094421 2024-04-13T03:51:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-13T03:51:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-04-13T03:51:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1058, 2024-04-13T03:51:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1005, 2024-04-13T03:51:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 52430935, 2024-04-13T03:51:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 412080, 2024-04-13T03:51:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 5, 2024-04-13T03:51:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 4, 2024-04-13T03:51:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.434012521, 2024-04-13T03:51:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.094395912, 2024-04-13T03:51:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.306356584, 2024-04-13T03:51:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.062378554, 2024-04-13T03:51:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1005, 2024-04-13T03:51:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.122409282, 2024-04-13T03:51:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000121800 2024-04-13T03:51:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1934864, 2024-04-13T03:51:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 206272 2024-04-13T03:51:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 383, 2024-04-13T03:51:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 629, 2024-04-13T03:51:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 46234388, 2024-04-13T03:51:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 711254, 2024-04-13T03:51:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 9, 2024-04-13T03:51:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 6, 2024-04-13T03:51:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.245921764, 2024-04-13T03:51:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.064804973, 2024-04-13T03:51:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.141272046, 2024-04-13T03:51:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.023393401, 2024-04-13T03:51:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 626, 2024-04-13T03:51:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.094819169, 2024-04-13T03:51:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000151468 2024-04-13T03:51:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-13T03:51:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 590096 2024-04-13T03:51:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-13T03:51:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-04-13T03:51:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-13T03:51:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-04-13T03:51:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-13T03:51:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-13T03:51:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 6644, 2024-04-13T03:51:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.358744090, 2024-04-13T03:51:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000053995 2024-04-13T03:51:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-13T03:51:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-13T03:51:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-13T03:51:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 0, 2024-04-13T03:51:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 0, 2024-04-13T03:51:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-04-13T03:51:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-04-13T03:51:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 0, 2024-04-13T03:51:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-04-13T03:51:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 0, 2024-04-13T03:51:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-04-13T03:51:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-04-13T03:51:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 11261, 2024-04-13T03:51:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 4, 2024-04-13T03:51:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 11257, 2024-04-13T03:51:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-04-13T03:51:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 11255, 2024-04-13T03:51:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 0, 2024-04-13T03:51:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 0, 2024-04-13T03:51:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 0, 2024-04-13T03:51:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 7, 2024-04-13T03:51:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 0, 2024-04-13T03:51:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 0, 2024-04-13T03:51:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 0, 2024-04-13T03:51:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 0, 2024-04-13T03:51:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 0, 2024-04-13T03:51:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-04-13T03:51:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-04-13T03:51:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 0, 2024-04-13T03:51:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-04-13T03:51:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 0, 2024-04-13T03:51:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 0, 2024-04-13T03:51:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 0, 2024-04-13T03:51:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 0, 2024-04-13T03:51:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 0, 2024-04-13T03:51:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-04-13T03:51:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 0, 2024-04-13T03:51:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 0, 2024-04-13T03:51:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 0, 2024-04-13T03:51:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 0, 2024-04-13T03:51:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 0, 2024-04-13T03:51:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 0 2024-04-13T03:51:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-04-13T03:51:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-04-13T03:51:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 0, 2024-04-13T03:51:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 0, 2024-04-13T03:51:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 0, 2024-04-13T03:51:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 0, 2024-04-13T03:51:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-04-13T03:51:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-04-13T03:51:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 0, 2024-04-13T03:51:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 0, 2024-04-13T03:51:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-04-13T03:51:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-04-13T03:51:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-04-13T03:51:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 0, 2024-04-13T03:51:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-04-13T03:51:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-04-13T03:51:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0 2024-04-13T03:51:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-04-13T03:51:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "evlrg": 0, 2024-04-13T03:51:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 0, 2024-04-13T03:51:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-04-13T03:51:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-04-13T03:51:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 50591, 2024-04-13T03:51:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-04-13T03:51:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-04-13T03:51:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 0, 2024-04-13T03:51:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 71, 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "segmjr": 6, 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 152619992, 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 152612606, 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-04-13T03:51:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 50591 2024-04-13T03:51:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-04-13T03:51:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 665, 2024-04-13T03:51:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 11302, 2024-04-13T03:51:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 10637, 2024-04-13T03:51:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 45, 2024-04-13T03:51:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 71, 2024-04-13T03:51:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 26, 2024-04-13T03:51:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 11269, 2024-04-13T03:51:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 11481, 2024-04-13T03:51:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 212, 2024-04-13T03:51:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 0, 2024-04-13T03:51:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 0, 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 0, 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 64232, 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 215416 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 0, 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 0, 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 0, 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 0, 2024-04-13T03:51:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 0, 2024-04-13T03:51:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-04-13T03:51:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-04-13T03:51:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-04-13T03:51:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-04-13T03:51:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-04-13T03:51:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-04-13T03:51:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-04-13T03:51:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-04-13T03:51:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-04-13T03:51:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-04-13T03:51:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-04-13T03:51:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-04-13T03:51:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-04-13T03:51:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-04-13T03:51:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-04-13T03:51:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-04-13T03:51:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-04-13T03:51:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 2, 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 5, 2024-04-13T03:51:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 3, 2024-04-13T03:51:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 2, 2024-04-13T03:51:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-04-13T03:51:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 0, 2024-04-13T03:51:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 0, 2024-04-13T03:51:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 5074, 2024-04-13T03:51:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-13T03:51:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-13T03:51:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 259812, 2024-04-13T03:51:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 805, 2024-04-13T03:51:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-13T03:51:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-13T03:51:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-13T03:51:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-13T03:51:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 186, 2024-04-13T03:51:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-13T03:51:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 9717781, 2024-04-13T03:51:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 82232, 2024-04-13T03:51:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-13T03:51:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-13T03:51:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-04-13T03:51:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-04-13T03:51:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-04-13T03:51:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 3997, 2024-04-13T03:51:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 21, 2024-04-13T03:51:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 3, 2024-04-13T03:51:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 3997, 2024-04-13T03:51:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-04-13T03:51:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3997, 2024-04-13T03:51:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 6.777895302, 2024-04-13T03:51:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001695745 2024-04-13T03:51:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-04-13T03:51:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-04-13T03:51:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 3997, 2024-04-13T03:51:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 3998 2024-04-13T03:51:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 3997, 2024-04-13T03:51:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 3997, 2024-04-13T03:51:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 0, 2024-04-13T03:51:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-04-13T03:51:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-04-13T03:51:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 2306, 2024-04-13T03:51:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 0, 2024-04-13T03:51:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 1689, 2024-04-13T03:51:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 0, 2024-04-13T03:51:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 0, 2024-04-13T03:51:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-04-13T03:51:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-04-13T03:51:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-04-13T03:51:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-04-13T03:51:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 0, 2024-04-13T03:51:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-04-13T03:51:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 1, 2024-04-13T03:51:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 0, 2024-04-13T03:51:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-04-13T03:51:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-04-13T03:51:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-04-13T03:51:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-04-13T03:51:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-04-13T03:51:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-04-13T03:51:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-04-13T03:51:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-04-13T03:51:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-04-13T03:51:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-04-13T03:51:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-04-13T03:51:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-04-13T03:51:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-04-13T03:51:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-04-13T03:51:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-04-13T03:51:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-04-13T03:51:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-04-13T03:51:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-04-13T03:51:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-04-13T03:51:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-04-13T03:51:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-04-13T03:51:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 117, 2024-04-13T03:51:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-04-13T03:51:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 27, 2024-04-13T03:51:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 10, 2024-04-13T03:51:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 11, 2024-04-13T03:51:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 1, 2024-04-13T03:51:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-04-13T03:51:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 0, 2024-04-13T03:51:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 2, 2024-04-13T03:51:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 0 2024-04-13T03:51:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-04-13T03:51:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 0, 2024-04-13T03:51:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-04-13T03:51:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-04-13T03:51:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "osd-slow-ops": { 2024-04-13T03:51:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 0 2024-04-13T03:51:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-04-13T03:51:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 0, 2024-04-13T03:51:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-04-13T03:51:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 0, 2024-04-13T03:51:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed_ops": 0, 2024-04-13T03:51:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 0, 2024-04-13T03:51:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-13T03:51:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 4406, 2024-04-13T03:51:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 151049934, 2024-04-13T03:51:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 4406, 2024-04-13T03:51:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-13T03:51:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-13T03:51:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 4406, 2024-04-13T03:51:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 151049934, 2024-04-13T03:51:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-13T03:51:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-13T03:51:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-13T03:51:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 3997, 2024-04-13T03:51:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 148425786, 2024-04-13T03:51:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 3997, 2024-04-13T03:51:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 148425786, 2024-04-13T03:51:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-04-13T03:51:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-13T03:51:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-13T03:51:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 3997, 2024-04-13T03:51:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 3997, 2024-04-13T03:51:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 3997, 2024-04-13T03:51:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 3997, 2024-04-13T03:51:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-13T03:51:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-13T03:51:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-13T03:51:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-13T03:51:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-13T03:51:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-04-13T03:51:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-04-13T03:51:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x556e68b70780": { 2024-04-13T03:51:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-13T03:51:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-13T03:51:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-13T03:51:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-13T03:51:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-13T03:51:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-04-13T03:51:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-04-13T03:51:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:24.929 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:51:25.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:24 smithi119 ceph-mon[35835]: pgmap v524: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.5 MiB/s wr, 49 op/s 2024-04-13T03:51:25.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:24 smithi016 ceph-mon[31772]: pgmap v524: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.5 MiB/s wr, 49 op/s 2024-04-13T03:51:25.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:24 smithi188 ceph-mon[36398]: pgmap v524: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 1.5 MiB/s wr, 49 op/s 2024-04-13T03:51:25.917 INFO:tasks.check_counter:Daemon mds.d mds.exported=0 2024-04-13T03:51:25.917 INFO:tasks.check_counter:Daemon mds.d mds.imported=0 2024-04-13T03:51:25.917 INFO:tasks.check_counter:Daemon mds.d mds_cache.dir_update=0 2024-04-13T03:51:25.917 INFO:tasks.check_counter:Daemon mds.d mds_cache.dir_update_receipt=0 2024-04-13T03:51:25.918 INFO:tasks.check_counter:Daemon mds.d mds.root_rsnaps=0 2024-04-13T03:51:25.918 INFO:tasks.check_counter:Daemon mds.d mds_server.req_mksnap_latency.avgcount=0 2024-04-13T03:51:25.918 INFO:tasks.check_counter:Daemon mds.d mds_server.req_rmsnap_latency.avgcount=0 2024-04-13T03:51:25.918 DEBUG:tasks.check_counter:Getting stats from g 2024-04-13T03:51:25.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-13T03:51:26.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:25 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:51:26.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:25 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:51:26.374 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:25 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:51:26.427 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:51:27.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:26 smithi119 ceph-mon[35835]: pgmap v525: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.5 MiB/s wr, 42 op/s 2024-04-13T03:51:27.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:26 smithi016 ceph-mon[31772]: pgmap v525: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.5 MiB/s wr, 42 op/s 2024-04-13T03:51:27.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:26 smithi188 ceph-mon[36398]: pgmap v525: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.5 MiB/s wr, 42 op/s 2024-04-13T03:51:28.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:27 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:28.200 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:27 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:51:28.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:27 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:51:28.201 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:27 smithi016 ceph-mon[31772]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:51:28.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:27 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:28.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:27 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:51:28.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:27 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:51:28.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:27 smithi119 ceph-mon[35835]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:51:28.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:27 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:28.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:27 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:51:28.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:27 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:51:28.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:27 smithi188 ceph-mon[36398]: from='mgr.25450 ' entity='mgr.x' 2024-04-13T03:51:28.786 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:51:28.786 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:28.786 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 332, 2024-04-13T03:51:28.787 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 321, 2024-04-13T03:51:28.787 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 1621120, 2024-04-13T03:51:28.787 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 198320, 2024-04-13T03:51:28.787 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 4, 2024-04-13T03:51:28.787 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 0, 2024-04-13T03:51:28.787 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.144152236, 2024-04-13T03:51:28.787 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.053585283, 2024-04-13T03:51:28.787 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.091882062, 2024-04-13T03:51:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.015820366, 2024-04-13T03:51:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 320, 2024-04-13T03:51:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.034244146, 2024-04-13T03:51:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000107012 2024-04-13T03:51:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:28.789 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:28.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.789 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1621120, 2024-04-13T03:51:28.789 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 198320 2024-04-13T03:51:28.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.789 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:28.790 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 7, 2024-04-13T03:51:28.790 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 248, 2024-04-13T03:51:28.790 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 3504, 2024-04-13T03:51:28.790 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 278320, 2024-04-13T03:51:28.790 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 6, 2024-04-13T03:51:28.790 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 3, 2024-04-13T03:51:28.790 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.051468610, 2024-04-13T03:51:28.790 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.038681982, 2024-04-13T03:51:28.790 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.005234135, 2024-04-13T03:51:28.791 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-13T03:51:28.791 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:28.791 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 246, 2024-04-13T03:51:28.791 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.045573731, 2024-04-13T03:51:28.791 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000185259 2024-04-13T03:51:28.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.791 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:28.792 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:28.792 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:28.792 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:28.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.792 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-04-13T03:51:28.792 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 278320 2024-04-13T03:51:28.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.792 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:28.793 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 0, 2024-04-13T03:51:28.793 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2, 2024-04-13T03:51:28.793 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 0, 2024-04-13T03:51:28.793 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 0, 2024-04-13T03:51:28.793 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 2, 2024-04-13T03:51:28.793 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 1, 2024-04-13T03:51:28.793 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.001091507, 2024-04-13T03:51:28.793 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.000000000, 2024-04-13T03:51:28.793 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.000896246, 2024-04-13T03:51:28.794 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-13T03:51:28.794 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:28.794 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:28.794 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:28.794 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:28.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.794 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:28.794 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:28.794 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:28.795 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:28.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.795 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-13T03:51:28.795 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-04-13T03:51:28.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.795 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-13T03:51:28.795 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 0, 2024-04-13T03:51:28.795 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-13T03:51:28.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.796 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-13T03:51:28.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:28.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:28.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:28.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:28.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:28.796 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:28.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:28.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:28.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:28.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:28.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:28.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:28.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:28.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:28.797 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:28.798 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:28.798 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:28.798 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:28.798 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:28.798 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:28.798 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:28.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:28.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:28.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:28.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:28.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:28.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:28.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:28.799 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-13T03:51:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:28.800 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:28.801 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 148961, 2024-04-13T03:51:28.801 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 29, 2024-04-13T03:51:28.801 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-13T03:51:28.801 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-13T03:51:28.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-13T03:51:28.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-13T03:51:28.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:28.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:28.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:28.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:28.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 0, 2024-04-13T03:51:28.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 0, 2024-04-13T03:51:28.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:28.803 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:28.803 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:28.803 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-13T03:51:28.803 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 928, 2024-04-13T03:51:28.803 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 2, 2024-04-13T03:51:28.803 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:28.803 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-13T03:51:28.804 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:28.804 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-13T03:51:28.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:28.804 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:28.804 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:28.804 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-13T03:51:28.804 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:28.804 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 335, 2024-04-13T03:51:28.804 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 1574274, 2024-04-13T03:51:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 335, 2024-04-13T03:51:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-13T03:51:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-13T03:51:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 335, 2024-04-13T03:51:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 1574274, 2024-04-13T03:51:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:28.806 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:28.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:28.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:28.806 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:51:29.273 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:28 smithi188 ceph-mon[36398]: pgmap v526: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.9 MiB/s wr, 73 op/s 2024-04-13T03:51:29.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:28 smithi119 ceph-mon[35835]: pgmap v526: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.9 MiB/s wr, 73 op/s 2024-04-13T03:51:29.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:28 smithi016 ceph-mon[31772]: pgmap v526: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.9 MiB/s wr, 73 op/s 2024-04-13T03:51:30.049 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-13T03:51:30.049 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-13T03:51:30.049 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-04-13T03:51:30.049 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-04-13T03:51:30.049 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-13T03:51:30.050 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-13T03:51:30.050 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-13T03:51:30.050 DEBUG:tasks.check_counter:Getting stats from j 2024-04-13T03:51:30.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-13T03:51:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.a/config 2024-04-13T03:51:31.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:30 smithi119 ceph-mon[35835]: pgmap v527: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 1.6 MiB/s wr, 55 op/s 2024-04-13T03:51:31.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:30 smithi016 ceph-mon[31772]: pgmap v527: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 1.6 MiB/s wr, 55 op/s 2024-04-13T03:51:31.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:30 smithi188 ceph-mon[36398]: pgmap v527: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 1.6 MiB/s wr, 55 op/s 2024-04-13T03:51:32.925 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-13T03:51:32.925 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:32.925 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 833, 2024-04-13T03:51:32.925 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 780, 2024-04-13T03:51:32.925 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 43996964, 2024-04-13T03:51:32.925 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 341950, 2024-04-13T03:51:32.925 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 7, 2024-04-13T03:51:32.926 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 2, 2024-04-13T03:51:32.926 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.332750578, 2024-04-13T03:51:32.926 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.077948206, 2024-04-13T03:51:32.926 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.234684701, 2024-04-13T03:51:32.926 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.040510387, 2024-04-13T03:51:32.926 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:32.926 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 779, 2024-04-13T03:51:32.926 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.098088908, 2024-04-13T03:51:32.926 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000125916 2024-04-13T03:51:32.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.927 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:32.927 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.927 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.927 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.927 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 1946496, 2024-04-13T03:51:32.927 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 208384 2024-04-13T03:51:32.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.928 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:32.928 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 1317, 2024-04-13T03:51:32.928 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 1563, 2024-04-13T03:51:32.928 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 63825408, 2024-04-13T03:51:32.928 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 1009415, 2024-04-13T03:51:32.928 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 8, 2024-04-13T03:51:32.928 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 6, 2024-04-13T03:51:32.928 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.539239374, 2024-04-13T03:51:32.929 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.146985266, 2024-04-13T03:51:32.929 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.270022118, 2024-04-13T03:51:32.929 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.086625095, 2024-04-13T03:51:32.929 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:32.929 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 1561, 2024-04-13T03:51:32.929 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.190366823, 2024-04-13T03:51:32.929 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000121951 2024-04-13T03:51:32.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.929 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:32.930 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.930 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.930 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.930 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-13T03:51:32.930 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 592512 2024-04-13T03:51:32.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.930 INFO:teuthology.orchestra.run.smithi016.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:32.930 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_messages": 2388, 2024-04-13T03:51:32.931 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages": 2390, 2024-04-13T03:51:32.931 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_bytes": 46840465, 2024-04-13T03:51:32.931 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_bytes": 759442, 2024-04-13T03:51:32.931 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_created_connections": 6, 2024-04-13T03:51:32.931 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_active_connections": 5, 2024-04-13T03:51:32.931 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_total_time": 0.681914170, 2024-04-13T03:51:32.931 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_send_time": 0.220068355, 2024-04-13T03:51:32.931 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_recv_time": 0.270053336, 2024-04-13T03:51:32.931 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_running_fast_dispatch_time": 0.141710116, 2024-04-13T03:51:32.932 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:32.932 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 2388, 2024-04-13T03:51:32.932 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.230135165, 2024-04-13T03:51:32.932 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000096371 2024-04-13T03:51:32.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.932 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:32.932 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.932 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.932 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.933 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-13T03:51:32.933 INFO:teuthology.orchestra.run.smithi016.stdout: "msgr_send_encrypted_bytes": 0 2024-04-13T03:51:32.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.933 INFO:teuthology.orchestra.run.smithi016.stdout: "cct": { 2024-04-13T03:51:32.933 INFO:teuthology.orchestra.run.smithi016.stdout: "total_workers": 1, 2024-04-13T03:51:32.933 INFO:teuthology.orchestra.run.smithi016.stdout: "unhealthy_workers": 0 2024-04-13T03:51:32.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.933 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-MDSRank": { 2024-04-13T03:51:32.934 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-13T03:51:32.934 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-13T03:51:32.934 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 6918, 2024-04-13T03:51:32.934 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.370689190, 2024-04-13T03:51:32.934 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000053583 2024-04-13T03:51:32.934 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:32.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: "queue_len": 0, 2024-04-13T03:51:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: "complete_latency": { 2024-04-13T03:51:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": { 2024-04-13T03:51:32.936 INFO:teuthology.orchestra.run.smithi016.stdout: "request": 0, 2024-04-13T03:51:32.936 INFO:teuthology.orchestra.run.smithi016.stdout: "reply": 0, 2024-04-13T03:51:32.936 INFO:teuthology.orchestra.run.smithi016.stdout: "reply_latency": { 2024-04-13T03:51:32.936 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.936 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.936 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.936 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_reply": 0, 2024-04-13T03:51:32.936 INFO:teuthology.orchestra.run.smithi016.stdout: "forward": 0, 2024-04-13T03:51:32.937 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_complete": 0, 2024-04-13T03:51:32.937 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:32.937 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_commit": 0, 2024-04-13T03:51:32.937 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_split": 0, 2024-04-13T03:51:32.937 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_merge": 0, 2024-04-13T03:51:32.937 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes": 10210, 2024-04-13T03:51:32.937 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_top": 12, 2024-04-13T03:51:32.937 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_bottom": 10198, 2024-04-13T03:51:32.937 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pin_tail": 0, 2024-04-13T03:51:32.938 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_pinned": 10193, 2024-04-13T03:51:32.938 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_expired": 0, 2024-04-13T03:51:32.938 INFO:teuthology.orchestra.run.smithi016.stdout: "inodes_with_caps": 0, 2024-04-13T03:51:32.938 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": 0, 2024-04-13T03:51:32.938 INFO:teuthology.orchestra.run.smithi016.stdout: "subtrees": 6, 2024-04-13T03:51:32.938 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse": 0, 2024-04-13T03:51:32.938 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_hit": 0, 2024-04-13T03:51:32.938 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_forward": 0, 2024-04-13T03:51:32.938 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_discover": 0, 2024-04-13T03:51:32.939 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:32.939 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:32.939 INFO:teuthology.orchestra.run.smithi016.stdout: "traverse_lock": 0, 2024-04-13T03:51:32.939 INFO:teuthology.orchestra.run.smithi016.stdout: "load_cent": 0, 2024-04-13T03:51:32.939 INFO:teuthology.orchestra.run.smithi016.stdout: "q": 0, 2024-04-13T03:51:32.939 INFO:teuthology.orchestra.run.smithi016.stdout: "exported": 0, 2024-04-13T03:51:32.939 INFO:teuthology.orchestra.run.smithi016.stdout: "exported_inodes": 0, 2024-04-13T03:51:32.939 INFO:teuthology.orchestra.run.smithi016.stdout: "imported": 0, 2024-04-13T03:51:32.939 INFO:teuthology.orchestra.run.smithi016.stdout: "imported_inodes": 0, 2024-04-13T03:51:32.940 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:32.940 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:32.940 INFO:teuthology.orchestra.run.smithi016.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:32.940 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rfiles": 0, 2024-04-13T03:51:32.940 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rbytes": 0, 2024-04-13T03:51:32.940 INFO:teuthology.orchestra.run.smithi016.stdout: "root_rsnaps": 0, 2024-04-13T03:51:32.940 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:32.940 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:32.940 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:32.941 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:32.941 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:32.941 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:32.941 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:32.941 INFO:teuthology.orchestra.run.smithi016.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:32.941 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_inode_file_caps": 0, 2024-04-13T03:51:32.941 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_revoke": 0, 2024-04-13T03:51:32.941 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_grant": 0, 2024-04-13T03:51:32.941 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:32.942 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:32.942 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:32.942 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps": 0, 2024-04-13T03:51:32.942 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_caps_dirty": 0, 2024-04-13T03:51:32.942 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_cap_release": 0, 2024-04-13T03:51:32.942 INFO:teuthology.orchestra.run.smithi016.stdout: "process_request_cap_release": 0 2024-04-13T03:51:32.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.942 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache": { 2024-04-13T03:51:32.942 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update": 0, 2024-04-13T03:51:32.943 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_update_receipt": 0, 2024-04-13T03:51:32.943 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_try_discover": 0, 2024-04-13T03:51:32.943 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_send_discover": 0, 2024-04-13T03:51:32.943 INFO:teuthology.orchestra.run.smithi016.stdout: "dir_handle_discover": 0, 2024-04-13T03:51:32.943 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays": 0, 2024-04-13T03:51:32.943 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_delayed": 0, 2024-04-13T03:51:32.943 INFO:teuthology.orchestra.run.smithi016.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:32.943 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_created": 0, 2024-04-13T03:51:32.943 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_enqueued": 0, 2024-04-13T03:51:32.944 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:32.944 INFO:teuthology.orchestra.run.smithi016.stdout: "strays_migrated": 0, 2024-04-13T03:51:32.944 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:32.944 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:32.944 INFO:teuthology.orchestra.run.smithi016.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:32.944 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_started": 0, 2024-04-13T03:51:32.944 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_completed": 0, 2024-04-13T03:51:32.944 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:32.944 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_exportdir": 0, 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_flush": 0, 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragmentdir": 0, 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: "ireq_inodestats": 0 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log": { 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: "evlrg": 0, 2024-04-13T03:51:32.945 INFO:teuthology.orchestra.run.smithi016.stdout: "evadd": 0, 2024-04-13T03:51:32.946 INFO:teuthology.orchestra.run.smithi016.stdout: "evex": 0, 2024-04-13T03:51:32.946 INFO:teuthology.orchestra.run.smithi016.stdout: "evtrm": 0, 2024-04-13T03:51:32.946 INFO:teuthology.orchestra.run.smithi016.stdout: "ev": 51214, 2024-04-13T03:51:32.946 INFO:teuthology.orchestra.run.smithi016.stdout: "evexg": 0, 2024-04-13T03:51:32.946 INFO:teuthology.orchestra.run.smithi016.stdout: "evexd": 0, 2024-04-13T03:51:32.946 INFO:teuthology.orchestra.run.smithi016.stdout: "segadd": 0, 2024-04-13T03:51:32.946 INFO:teuthology.orchestra.run.smithi016.stdout: "segex": 0, 2024-04-13T03:51:32.946 INFO:teuthology.orchestra.run.smithi016.stdout: "segtrm": 0, 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "seg": 74, 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "segmjr": 6, 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "segexg": 0, 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "segexd": 0, 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "expos": 4194304, 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "wrpos": 155760358, 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "rdpos": 155747864, 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "jlat": { 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: "replayed": 51214 2024-04-13T03:51:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mem": { 2024-04-13T03:51:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: "ino": 556, 2024-04-13T03:51:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: "ino+": 10311, 2024-04-13T03:51:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: "ino-": 9755, 2024-04-13T03:51:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: "dir": 41, 2024-04-13T03:51:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: "dir+": 93, 2024-04-13T03:51:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: "dir-": 52, 2024-04-13T03:51:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: "dn": 10210, 2024-04-13T03:51:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: "dn+": 10574, 2024-04-13T03:51:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: "dn-": 364, 2024-04-13T03:51:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: "cap": 0, 2024-04-13T03:51:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: "cap+": 0, 2024-04-13T03:51:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: "cap-": 0, 2024-04-13T03:51:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: "rss": 63476, 2024-04-13T03:51:32.950 INFO:teuthology.orchestra.run.smithi016.stdout: "heap": 215416 2024-04-13T03:51:32.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.950 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_server": { 2024-04-13T03:51:32.950 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_client_request": 0, 2024-04-13T03:51:32.950 INFO:teuthology.orchestra.run.smithi016.stdout: "dispatch_server_request": 0, 2024-04-13T03:51:32.950 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_request": 0, 2024-04-13T03:51:32.950 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_client_session": 0, 2024-04-13T03:51:32.950 INFO:teuthology.orchestra.run.smithi016.stdout: "handle_peer_request": 0, 2024-04-13T03:51:32.950 INFO:teuthology.orchestra.run.smithi016.stdout: "req_create_latency": { 2024-04-13T03:51:32.951 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.951 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.951 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.951 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getattr_latency": { 2024-04-13T03:51:32.951 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.951 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.951 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: "req_link_latency": { 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.953 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookup_latency": { 2024-04-13T03:51:32.953 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.953 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.953 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.953 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:32.953 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.953 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.953 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.954 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupino_latency": { 2024-04-13T03:51:32.954 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.954 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.954 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.954 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupname_latency": { 2024-04-13T03:51:32.954 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.954 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:32.955 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.956 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.956 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.956 INFO:teuthology.orchestra.run.smithi016.stdout: "req_lssnap_latency": { 2024-04-13T03:51:32.956 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.956 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.956 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.957 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mkdir_latency": { 2024-04-13T03:51:32.957 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.957 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mknod_latency": { 2024-04-13T03:51:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: "req_mksnap_latency": { 2024-04-13T03:51:32.958 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.959 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.959 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.959 INFO:teuthology.orchestra.run.smithi016.stdout: "req_open_latency": { 2024-04-13T03:51:32.959 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.959 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.959 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.959 INFO:teuthology.orchestra.run.smithi016.stdout: "req_readdir_latency": { 2024-04-13T03:51:32.960 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.960 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.960 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.960 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rename_latency": { 2024-04-13T03:51:32.960 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.960 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.960 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.961 INFO:teuthology.orchestra.run.smithi016.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:32.961 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.961 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.961 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.961 INFO:teuthology.orchestra.run.smithi016.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:32.961 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.961 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.961 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.962 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmdir_latency": { 2024-04-13T03:51:32.962 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.962 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.962 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.962 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:32.962 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.962 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.963 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.963 INFO:teuthology.orchestra.run.smithi016.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:32.963 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.963 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.963 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.963 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setattr_latency": { 2024-04-13T03:51:32.963 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.964 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.964 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.964 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:32.964 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.964 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.964 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.964 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:32.965 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.965 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.965 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.965 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setlayout_latency": { 2024-04-13T03:51:32.965 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.965 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.965 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.966 INFO:teuthology.orchestra.run.smithi016.stdout: "req_setxattr_latency": { 2024-04-13T03:51:32.966 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.966 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.966 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.966 INFO:teuthology.orchestra.run.smithi016.stdout: "req_symlink_latency": { 2024-04-13T03:51:32.966 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.966 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.966 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.967 INFO:teuthology.orchestra.run.smithi016.stdout: "req_unlink_latency": { 2024-04-13T03:51:32.967 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.967 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.967 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.967 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:32.967 INFO:teuthology.orchestra.run.smithi016.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:32.967 INFO:teuthology.orchestra.run.smithi016.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:32.968 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.968 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.968 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:32.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.968 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessions": { 2024-04-13T03:51:32.968 INFO:teuthology.orchestra.run.smithi016.stdout: "session_count": 2, 2024-04-13T03:51:32.968 INFO:teuthology.orchestra.run.smithi016.stdout: "session_add": 5, 2024-04-13T03:51:32.968 INFO:teuthology.orchestra.run.smithi016.stdout: "session_remove": 3, 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_open": 4, 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: "sessions_stale": 0, 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: "total_load": 0, 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: "average_load": 0, 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: "avg_session_uptime": 3725, 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool": { 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:32.969 INFO:teuthology.orchestra.run.smithi016.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:32.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:32.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:32.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:32.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:32.970 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:32.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:32.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:32.971 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:32.974 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:32.974 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:32.974 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:32.974 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:32.974 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:32.974 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:32.974 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:32.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:32.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:32.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:32.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:32.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_items": 0, 2024-04-13T03:51:32.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:32.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:32.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:32.976 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_bytes": 247642, 2024-04-13T03:51:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_anon_items": 691, 2024-04-13T03:51:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_bytes": 0, 2024-04-13T03:51:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: "buffer_meta_items": 0, 2024-04-13T03:51:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bytes": 0, 2024-04-13T03:51:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_items": 0, 2024-04-13T03:51:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:32.978 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:32.978 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:32.978 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_items": 186, 2024-04-13T03:51:32.978 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:32.978 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:32.978 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:32.978 INFO:teuthology.orchestra.run.smithi016.stdout: "pgmap_items": 0, 2024-04-13T03:51:32.978 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_bytes": 8722299, 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_co_items": 70869, 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_1_items": 0, 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: "unittest_2_items": 0 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter": { 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: "op_active": 0, 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: "op_laggy": 0, 2024-04-13T03:51:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send": 4119, 2024-04-13T03:51:32.980 INFO:teuthology.orchestra.run.smithi016.stdout: "op_send_bytes": 21, 2024-04-13T03:51:32.980 INFO:teuthology.orchestra.run.smithi016.stdout: "op_resend": 3, 2024-04-13T03:51:32.980 INFO:teuthology.orchestra.run.smithi016.stdout: "op_reply": 4119, 2024-04-13T03:51:32.980 INFO:teuthology.orchestra.run.smithi016.stdout: "op_latency": { 2024-04-13T03:51:32.980 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 4119, 2024-04-13T03:51:32.980 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 7.014891688, 2024-04-13T03:51:32.980 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.001703056 2024-04-13T03:51:32.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.980 INFO:teuthology.orchestra.run.smithi016.stdout: "op_inflight": 0, 2024-04-13T03:51:32.981 INFO:teuthology.orchestra.run.smithi016.stdout: "oplen_avg": { 2024-04-13T03:51:32.981 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 4119, 2024-04-13T03:51:32.981 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 4120 2024-04-13T03:51:32.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.981 INFO:teuthology.orchestra.run.smithi016.stdout: "op": 4119, 2024-04-13T03:51:32.981 INFO:teuthology.orchestra.run.smithi016.stdout: "op_r": 4119, 2024-04-13T03:51:32.981 INFO:teuthology.orchestra.run.smithi016.stdout: "op_w": 0, 2024-04-13T03:51:32.981 INFO:teuthology.orchestra.run.smithi016.stdout: "op_rmw": 0, 2024-04-13T03:51:32.982 INFO:teuthology.orchestra.run.smithi016.stdout: "op_pg": 0, 2024-04-13T03:51:32.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_stat": 2316, 2024-04-13T03:51:32.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_create": 0, 2024-04-13T03:51:32.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_read": 1801, 2024-04-13T03:51:32.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_write": 0, 2024-04-13T03:51:32.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writefull": 0, 2024-04-13T03:51:32.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_writesame": 0, 2024-04-13T03:51:32.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_append": 0, 2024-04-13T03:51:32.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_zero": 0, 2024-04-13T03:51:32.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_truncate": 0, 2024-04-13T03:51:32.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_delete": 0, 2024-04-13T03:51:32.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_mapext": 0, 2024-04-13T03:51:32.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:32.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:32.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_getxattr": 1, 2024-04-13T03:51:32.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_setxattr": 0, 2024-04-13T03:51:32.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:32.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_call": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_watch": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_notify": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "osdop_other": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_active": 0, 2024-04-13T03:51:32.984 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_send": 0, 2024-04-13T03:51:32.985 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_resend": 0, 2024-04-13T03:51:32.985 INFO:teuthology.orchestra.run.smithi016.stdout: "linger_ping": 0, 2024-04-13T03:51:32.985 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_active": 0, 2024-04-13T03:51:32.985 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_send": 0, 2024-04-13T03:51:32.985 INFO:teuthology.orchestra.run.smithi016.stdout: "poolop_resend": 0, 2024-04-13T03:51:32.985 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_active": 0, 2024-04-13T03:51:32.985 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_send": 0, 2024-04-13T03:51:32.985 INFO:teuthology.orchestra.run.smithi016.stdout: "poolstat_resend": 0, 2024-04-13T03:51:32.985 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_active": 0, 2024-04-13T03:51:32.986 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_send": 0, 2024-04-13T03:51:32.986 INFO:teuthology.orchestra.run.smithi016.stdout: "statfs_resend": 0, 2024-04-13T03:51:32.986 INFO:teuthology.orchestra.run.smithi016.stdout: "command_active": 0, 2024-04-13T03:51:32.986 INFO:teuthology.orchestra.run.smithi016.stdout: "command_send": 0, 2024-04-13T03:51:32.986 INFO:teuthology.orchestra.run.smithi016.stdout: "command_resend": 0, 2024-04-13T03:51:32.986 INFO:teuthology.orchestra.run.smithi016.stdout: "map_epoch": 117, 2024-04-13T03:51:32.986 INFO:teuthology.orchestra.run.smithi016.stdout: "map_full": 0, 2024-04-13T03:51:32.986 INFO:teuthology.orchestra.run.smithi016.stdout: "map_inc": 27, 2024-04-13T03:51:32.986 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_sessions": 9, 2024-04-13T03:51:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_open": 10, 2024-04-13T03:51:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_session_close": 1, 2024-04-13T03:51:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_laggy": 0, 2024-04-13T03:51:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_wr": 0, 2024-04-13T03:51:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_rd": 2, 2024-04-13T03:51:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_del": 0 2024-04-13T03:51:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: "oft": { 2024-04-13T03:51:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_objs": 0, 2024-04-13T03:51:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_updates": 0, 2024-04-13T03:51:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "omap_total_removes": 0 2024-04-13T03:51:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "osd-slow-ops": { 2024-04-13T03:51:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "slow_ops_count": 0 2024-04-13T03:51:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "purge_queue": { 2024-04-13T03:51:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:32.989 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_ops_high_water": 0, 2024-04-13T03:51:32.989 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing": 0, 2024-04-13T03:51:32.989 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executing_high_water": 0, 2024-04-13T03:51:32.989 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed_ops": 0, 2024-04-13T03:51:32.989 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_executed": 0, 2024-04-13T03:51:32.989 INFO:teuthology.orchestra.run.smithi016.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:32.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.989 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:32.989 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:32.990 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-13T03:51:32.990 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:32.990 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 4534, 2024-04-13T03:51:32.990 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 154230737, 2024-04-13T03:51:32.990 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:32.990 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 4534, 2024-04-13T03:51:32.990 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-13T03:51:32.990 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-13T03:51:32.990 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 4534, 2024-04-13T03:51:32.991 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 154230737, 2024-04-13T03:51:32.991 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:32.991 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.991 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.991 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.991 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:32.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.991 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:32.992 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:32.992 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 104857600, 2024-04-13T03:51:32.992 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:32.992 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-13T03:51:32.992 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-13T03:51:32.992 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:32.992 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:32.992 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 4119, 2024-04-13T03:51:32.992 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 151573317, 2024-04-13T03:51:32.993 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 4119, 2024-04-13T03:51:32.993 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 151573317, 2024-04-13T03:51:32.993 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:32.993 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.993 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.993 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:32.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.993 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:32.994 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:32.994 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 1024, 2024-04-13T03:51:32.994 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:32.994 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-13T03:51:32.994 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-13T03:51:32.994 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:32.994 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:32.994 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 4119, 2024-04-13T03:51:32.994 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 4119, 2024-04-13T03:51:32.995 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 4119, 2024-04-13T03:51:32.995 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 4119, 2024-04-13T03:51:32.995 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:32.995 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.995 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.995 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:32.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.996 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:32.996 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:32.996 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-13T03:51:32.996 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:32.996 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-13T03:51:32.996 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-13T03:51:32.996 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:32.996 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:32.997 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-13T03:51:32.997 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-13T03:51:32.997 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-04-13T03:51:32.997 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-04-13T03:51:32.997 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:32.997 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.997 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.997 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:32.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-13T03:51:32.998 INFO:teuthology.orchestra.run.smithi016.stdout: "throttle-write_buf_throttle-0x564fa508e780": { 2024-04-13T03:51:32.998 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 0, 2024-04-13T03:51:32.998 INFO:teuthology.orchestra.run.smithi016.stdout: "max": 3758096384, 2024-04-13T03:51:32.998 INFO:teuthology.orchestra.run.smithi016.stdout: "get_started": 0, 2024-04-13T03:51:32.998 INFO:teuthology.orchestra.run.smithi016.stdout: "get": 0, 2024-04-13T03:51:32.998 INFO:teuthology.orchestra.run.smithi016.stdout: "get_sum": 0, 2024-04-13T03:51:32.998 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:32.998 INFO:teuthology.orchestra.run.smithi016.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:32.999 INFO:teuthology.orchestra.run.smithi016.stdout: "take": 0, 2024-04-13T03:51:32.999 INFO:teuthology.orchestra.run.smithi016.stdout: "take_sum": 0, 2024-04-13T03:51:32.999 INFO:teuthology.orchestra.run.smithi016.stdout: "put": 0, 2024-04-13T03:51:32.999 INFO:teuthology.orchestra.run.smithi016.stdout: "put_sum": 0, 2024-04-13T03:51:32.999 INFO:teuthology.orchestra.run.smithi016.stdout: "wait": { 2024-04-13T03:51:32.999 INFO:teuthology.orchestra.run.smithi016.stdout: "avgcount": 0, 2024-04-13T03:51:32.999 INFO:teuthology.orchestra.run.smithi016.stdout: "sum": 0.000000000, 2024-04-13T03:51:32.999 INFO:teuthology.orchestra.run.smithi016.stdout: "avgtime": 0.000000000 2024-04-13T03:51:32.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:33.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-13T03:51:33.000 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-13T03:51:33.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:33 smithi119 ceph-mon[35835]: pgmap v528: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 3.8 MiB/s wr, 126 op/s 2024-04-13T03:51:33.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:33 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:33.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:33 smithi016 ceph-mon[31772]: pgmap v528: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 3.8 MiB/s wr, 126 op/s 2024-04-13T03:51:33.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:33 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:33 smithi188 ceph-mon[36398]: pgmap v528: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 3.8 MiB/s wr, 126 op/s 2024-04-13T03:51:33.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:33 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:33.987 INFO:tasks.check_counter:Daemon mds.j mds.exported=0 2024-04-13T03:51:33.987 INFO:tasks.check_counter:Daemon mds.j mds.imported=0 2024-04-13T03:51:33.987 INFO:tasks.check_counter:Daemon mds.j mds_cache.dir_update=0 2024-04-13T03:51:33.987 INFO:tasks.check_counter:Daemon mds.j mds_cache.dir_update_receipt=0 2024-04-13T03:51:33.987 INFO:tasks.check_counter:Daemon mds.j mds.root_rsnaps=0 2024-04-13T03:51:33.987 INFO:tasks.check_counter:Daemon mds.j mds_server.req_mksnap_latency.avgcount=0 2024-04-13T03:51:33.987 INFO:tasks.check_counter:Daemon mds.j mds_server.req_rmsnap_latency.avgcount=0 2024-04-13T03:51:33.988 DEBUG:tasks.check_counter:Getting stats from b 2024-04-13T03:51:33.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-13T03:51:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:51:35.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:35 smithi119 ceph-mon[35835]: pgmap v529: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 2.9 MiB/s wr, 111 op/s 2024-04-13T03:51:35.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:35 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:35.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:35 smithi016 ceph-mon[31772]: pgmap v529: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 2.9 MiB/s wr, 111 op/s 2024-04-13T03:51:35.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:35 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:35.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:35 smithi188 ceph-mon[36398]: pgmap v529: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 2.9 MiB/s wr, 111 op/s 2024-04-13T03:51:35.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:35 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:36.287 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:36 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:36.288 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:36 smithi119 ceph-mon[35835]: pgmap v530: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 2.8 MiB/s wr, 109 op/s 2024-04-13T03:51:36.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:36 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:36.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:36 smithi016 ceph-mon[31772]: pgmap v530: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 2.8 MiB/s wr, 109 op/s 2024-04-13T03:51:36.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:36 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:36.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:36 smithi188 ceph-mon[36398]: pgmap v530: 129 pgs: 129 active+clean; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 2.8 MiB/s wr, 109 op/s 2024-04-13T03:51:36.929 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-13T03:51:36.929 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:36.929 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 21142, 2024-04-13T03:51:36.929 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 21171, 2024-04-13T03:51:36.929 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 7229405, 2024-04-13T03:51:36.929 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 179065027, 2024-04-13T03:51:36.930 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 19, 2024-04-13T03:51:36.930 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 6, 2024-04-13T03:51:36.930 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 2.389735314, 2024-04-13T03:51:36.930 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.624339211, 2024-04-13T03:51:36.930 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.444757963, 2024-04-13T03:51:36.930 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 1.217684602, 2024-04-13T03:51:36.930 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:36.930 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 21163, 2024-04-13T03:51:36.931 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.896049722, 2024-04-13T03:51:36.931 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000089592 2024-04-13T03:51:36.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.931 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:36.931 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 267, 2024-04-13T03:51:36.931 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000040164, 2024-04-13T03:51:36.931 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000150 2024-04-13T03:51:36.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.931 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 2085040, 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 703136 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 397318, 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 532366, 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 107961514, 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 429192732, 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 12, 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 6, 2024-04-13T03:51:36.932 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 25.585676862, 2024-04-13T03:51:36.933 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 11.478211671, 2024-04-13T03:51:36.933 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 15.587211450, 2024-04-13T03:51:36.933 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 1.598608937, 2024-04-13T03:51:36.933 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:36.933 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 532369, 2024-04-13T03:51:36.933 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 25.785428922, 2024-04-13T03:51:36.933 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000048435 2024-04-13T03:51:36.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.933 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:36.934 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 678819, 2024-04-13T03:51:36.934 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.144755065, 2024-04-13T03:51:36.934 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000213 2024-04-13T03:51:36.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.934 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 530144, 2024-04-13T03:51:36.934 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 397008 2024-04-13T03:51:36.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.934 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:36.934 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 229646, 2024-04-13T03:51:36.935 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 303482, 2024-04-13T03:51:36.935 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 53819933, 2024-04-13T03:51:36.935 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 289046842, 2024-04-13T03:51:36.935 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 12, 2024-04-13T03:51:36.935 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 7, 2024-04-13T03:51:36.935 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 17.027859714, 2024-04-13T03:51:36.935 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 7.546023837, 2024-04-13T03:51:36.935 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 9.845128099, 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 1.279653904, 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 303481, 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 12.109901082, 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000039903 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 487896, 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.084897758, 2024-04-13T03:51:36.936 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000174 2024-04-13T03:51:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 45248, 2024-04-13T03:51:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 1664 2024-04-13T03:51:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-04-13T03:51:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-04-13T03:51:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-04-13T03:51:36.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-04-13T03:51:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-13T03:51:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-13T03:51:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 66776, 2024-04-13T03:51:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 68.670262762, 2024-04-13T03:51:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001028367 2024-04-13T03:51:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.938 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:36.939 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-13T03:51:36.939 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-13T03:51:36.939 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 22450, 2024-04-13T03:51:36.939 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 4.186739081, 2024-04-13T03:51:36.939 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000186491 2024-04-13T03:51:36.939 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:36.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.939 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-04-13T03:51:36.939 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 112282, 2024-04-13T03:51:36.940 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 96959, 2024-04-13T03:51:36.940 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-04-13T03:51:36.940 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 96959, 2024-04-13T03:51:36.940 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 787.731970539, 2024-04-13T03:51:36.940 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.008124382 2024-04-13T03:51:36.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.940 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-04-13T03:51:36.940 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 15323, 2024-04-13T03:51:36.940 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 13, 2024-04-13T03:51:36.941 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:36.941 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 50, 2024-04-13T03:51:36.941 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 3, 2024-04-13T03:51:36.941 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 2, 2024-04-13T03:51:36.941 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 1668, 2024-04-13T03:51:36.941 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 308, 2024-04-13T03:51:36.941 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 788, 2024-04-13T03:51:36.941 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 572, 2024-04-13T03:51:36.941 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 1227, 2024-04-13T03:51:36.942 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 2213, 2024-04-13T03:51:36.942 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 288, 2024-04-13T03:51:36.942 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 288, 2024-04-13T03:51:36.942 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 17, 2024-04-13T03:51:36.942 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 174845, 2024-04-13T03:51:36.942 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 111201, 2024-04-13T03:51:36.942 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 15297, 2024-04-13T03:51:36.942 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 268, 2024-04-13T03:51:36.942 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:36.943 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:36.943 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 1251, 2024-04-13T03:51:36.943 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 26441, 2024-04-13T03:51:36.943 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-04-13T03:51:36.943 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 25, 2024-04-13T03:51:36.943 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 1398, 2024-04-13T03:51:36.943 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 14, 2024-04-13T03:51:36.943 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 603, 2024-04-13T03:51:36.943 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:36.944 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:36.944 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:36.944 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 747, 2024-04-13T03:51:36.944 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 212049091, 2024-04-13T03:51:36.944 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 6, 2024-04-13T03:51:36.944 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:36.944 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:36.944 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:36.944 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:36.945 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:36.945 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:36.945 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:36.945 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:36.945 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 28, 2024-04-13T03:51:36.945 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 27902, 2024-04-13T03:51:36.945 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 48681, 2024-04-13T03:51:36.945 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:36.945 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:36.946 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:36.946 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 45798, 2024-04-13T03:51:36.946 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 10842, 2024-04-13T03:51:36.946 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 4915, 2024-04-13T03:51:36.946 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 56263 2024-04-13T03:51:36.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.946 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-04-13T03:51:36.946 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 533, 2024-04-13T03:51:36.946 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 267, 2024-04-13T03:51:36.947 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 158, 2024-04-13T03:51:36.947 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 433, 2024-04-13T03:51:36.947 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 3668, 2024-04-13T03:51:36.947 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 70, 2024-04-13T03:51:36.947 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 21, 2024-04-13T03:51:36.947 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:36.947 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 11543, 2024-04-13T03:51:36.947 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 11473, 2024-04-13T03:51:36.948 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:36.948 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-04-13T03:51:36.948 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:36.948 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:36.948 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:36.948 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-04-13T03:51:36.948 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-04-13T03:51:36.948 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:36.948 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:36.949 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:36.949 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 27, 2024-04-13T03:51:36.949 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-04-13T03:51:36.949 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 5, 2024-04-13T03:51:36.949 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:36.949 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-04-13T03:51:36.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.949 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-04-13T03:51:36.949 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-04-13T03:51:36.950 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 76293, 2024-04-13T03:51:36.950 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 12291, 2024-04-13T03:51:36.950 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 12291, 2024-04-13T03:51:36.950 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 64002, 2024-04-13T03:51:36.950 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-04-13T03:51:36.950 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 12290, 2024-04-13T03:51:36.950 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 144, 2024-04-13T03:51:36.950 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 16, 2024-04-13T03:51:36.950 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 16, 2024-04-13T03:51:36.951 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 128, 2024-04-13T03:51:36.951 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 6, 2024-04-13T03:51:36.951 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-04-13T03:51:36.951 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 15, 2024-04-13T03:51:36.951 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 42951514, 2024-04-13T03:51:36.951 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 547392741, 2024-04-13T03:51:36.951 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-04-13T03:51:36.951 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-04-13T03:51:36.951 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 35077, 2024-04-13T03:51:36.952 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 198.608140678, 2024-04-13T03:51:36.952 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005662061 2024-04-13T03:51:36.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.952 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-04-13T03:51:36.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.952 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-04-13T03:51:36.952 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1133, 2024-04-13T03:51:36.952 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 13843, 2024-04-13T03:51:36.952 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 12710, 2024-04-13T03:51:36.953 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 141, 2024-04-13T03:51:36.953 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 383, 2024-04-13T03:51:36.953 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 242, 2024-04-13T03:51:36.953 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 1688, 2024-04-13T03:51:36.953 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 25518, 2024-04-13T03:51:36.953 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 23830, 2024-04-13T03:51:36.953 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 230, 2024-04-13T03:51:36.953 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 13061, 2024-04-13T03:51:36.953 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 12831, 2024-04-13T03:51:36.954 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 80344, 2024-04-13T03:51:36.954 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 231800 2024-04-13T03:51:36.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.954 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-04-13T03:51:36.954 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 160846, 2024-04-13T03:51:36.954 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 31770, 2024-04-13T03:51:36.954 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 112282, 2024-04-13T03:51:36.954 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 1527, 2024-04-13T03:51:36.954 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 63194, 2024-04-13T03:51:36.955 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-04-13T03:51:36.955 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 12284, 2024-04-13T03:51:36.955 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 93.633094675, 2024-04-13T03:51:36.955 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.007622361 2024-04-13T03:51:36.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.955 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-04-13T03:51:36.955 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 22967, 2024-04-13T03:51:36.955 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 211.039995291, 2024-04-13T03:51:36.955 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.009188835 2024-04-13T03:51:36.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.956 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:36.956 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.956 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.956 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.956 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-04-13T03:51:36.956 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.956 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.957 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.957 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-04-13T03:51:36.957 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 22691, 2024-04-13T03:51:36.957 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 230.262468981, 2024-04-13T03:51:36.957 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.010147744 2024-04-13T03:51:36.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.957 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:36.957 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.958 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.958 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.958 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-04-13T03:51:36.958 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.958 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.958 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.958 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-04-13T03:51:36.959 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.959 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.959 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.959 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:36.959 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.959 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.959 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.960 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:36.960 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.961 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.961 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.961 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-04-13T03:51:36.961 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 17, 2024-04-13T03:51:36.961 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.005152329, 2024-04-13T03:51:36.961 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000303078 2024-04-13T03:51:36.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.961 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-04-13T03:51:36.962 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 119, 2024-04-13T03:51:36.962 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.760798308, 2024-04-13T03:51:36.962 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.006393263 2024-04-13T03:51:36.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.962 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-04-13T03:51:36.962 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.962 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.962 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.963 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-04-13T03:51:36.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 17, 2024-04-13T03:51:36.963 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 6.910241631, 2024-04-13T03:51:36.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.406484801 2024-04-13T03:51:36.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.963 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-04-13T03:51:36.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 10, 2024-04-13T03:51:36.963 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.010495166, 2024-04-13T03:51:36.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001049516 2024-04-13T03:51:36.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.964 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-04-13T03:51:36.964 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 24829, 2024-04-13T03:51:36.964 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 143.814106243, 2024-04-13T03:51:36.964 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005792182 2024-04-13T03:51:36.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.964 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-04-13T03:51:36.964 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2130, 2024-04-13T03:51:36.965 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 21.214358563, 2024-04-13T03:51:36.965 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.009959792 2024-04-13T03:51:36.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.965 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:36.965 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.965 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.965 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.965 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:36.966 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.966 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.966 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.966 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-04-13T03:51:36.966 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1, 2024-04-13T03:51:36.966 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.001026432, 2024-04-13T03:51:36.966 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001026432 2024-04-13T03:51:36.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.967 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:36.967 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 11, 2024-04-13T03:51:36.967 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 3.619432162, 2024-04-13T03:51:36.967 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.329039287 2024-04-13T03:51:36.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.967 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:36.967 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.967 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.967 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.968 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-04-13T03:51:36.968 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 32, 2024-04-13T03:51:36.968 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.960832220, 2024-04-13T03:51:36.968 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.030026006 2024-04-13T03:51:36.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.968 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:36.968 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.968 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.969 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:36.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 300, 2024-04-13T03:51:36.969 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.952066158, 2024-04-13T03:51:36.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003173553 2024-04-13T03:51:36.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.969 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-04-13T03:51:36.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.970 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.970 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.970 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-04-13T03:51:36.970 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 8, 2024-04-13T03:51:36.970 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.021684772, 2024-04-13T03:51:36.970 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002710596 2024-04-13T03:51:36.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.970 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-04-13T03:51:36.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.971 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.971 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-04-13T03:51:36.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 11543, 2024-04-13T03:51:36.971 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 74.526217608, 2024-04-13T03:51:36.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.006456399 2024-04-13T03:51:36.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.972 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:36.972 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:36.972 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:36.972 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.972 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.972 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.972 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:36.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.972 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-04-13T03:51:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 2, 2024-04-13T03:51:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 6, 2024-04-13T03:51:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 4, 2024-04-13T03:51:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 2, 2024-04-13T03:51:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-04-13T03:51:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 23437, 2024-04-13T03:51:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 11718, 2024-04-13T03:51:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 1004, 2024-04-13T03:51:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-04-13T03:51:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:36.977 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:36.977 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:36.977 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:36.977 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:36.977 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:36.977 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:36.977 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-04-13T03:51:36.977 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:36.977 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:36.978 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:36.978 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:36.978 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 348304, 2024-04-13T03:51:36.978 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 961, 2024-04-13T03:51:36.978 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-04-13T03:51:36.978 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-04-13T03:51:36.978 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-04-13T03:51:36.978 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-04-13T03:51:36.978 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:36.979 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:36.979 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:36.979 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:36.979 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:36.979 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 186, 2024-04-13T03:51:36.979 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:36.979 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:36.979 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:36.979 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-04-13T03:51:36.980 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 4642894, 2024-04-13T03:51:36.980 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 52222, 2024-04-13T03:51:36.980 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:36.980 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-04-13T03:51:36.980 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:36.980 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-04-13T03:51:36.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.980 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-04-13T03:51:36.980 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-04-13T03:51:36.981 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-04-13T03:51:36.981 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 60949, 2024-04-13T03:51:36.981 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 545929320, 2024-04-13T03:51:36.981 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 37, 2024-04-13T03:51:36.981 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 60949, 2024-04-13T03:51:36.981 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-04-13T03:51:36.981 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 60949, 2024-04-13T03:51:36.981 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 433.403775131, 2024-04-13T03:51:36.981 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.007110925 2024-04-13T03:51:36.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.982 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-04-13T03:51:36.982 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-04-13T03:51:36.982 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 60949, 2024-04-13T03:51:36.982 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 61976 2024-04-13T03:51:36.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.982 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 60949, 2024-04-13T03:51:36.982 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 584, 2024-04-13T03:51:36.982 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 60365, 2024-04-13T03:51:36.983 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-04-13T03:51:36.983 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-04-13T03:51:36.983 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 0, 2024-04-13T03:51:36.983 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 51, 2024-04-13T03:51:36.983 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 565, 2024-04-13T03:51:36.983 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 35642, 2024-04-13T03:51:36.983 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 352, 2024-04-13T03:51:36.983 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-04-13T03:51:36.983 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-04-13T03:51:36.984 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-04-13T03:51:36.984 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 3, 2024-04-13T03:51:36.984 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 23612, 2024-04-13T03:51:36.984 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-04-13T03:51:36.984 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:36.984 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:36.984 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 0, 2024-04-13T03:51:36.984 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 98, 2024-04-13T03:51:36.984 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:36.985 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:36.985 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:36.985 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-04-13T03:51:36.985 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-04-13T03:51:36.985 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-04-13T03:51:36.985 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:36.985 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-04-13T03:51:36.985 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:36.985 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-04-13T03:51:36.986 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-04-13T03:51:36.986 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-04-13T03:51:36.986 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-04-13T03:51:36.986 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-04-13T03:51:36.986 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-04-13T03:51:36.986 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-04-13T03:51:36.986 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-04-13T03:51:36.986 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-04-13T03:51:36.986 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-04-13T03:51:36.987 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-04-13T03:51:36.987 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-04-13T03:51:36.987 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-04-13T03:51:36.987 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-04-13T03:51:36.987 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-04-13T03:51:36.987 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-04-13T03:51:36.987 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-04-13T03:51:36.987 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 117, 2024-04-13T03:51:36.987 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-04-13T03:51:36.988 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 32, 2024-04-13T03:51:36.988 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-04-13T03:51:36.988 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-04-13T03:51:36.988 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-04-13T03:51:36.988 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-04-13T03:51:36.988 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 1279, 2024-04-13T03:51:36.988 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 32, 2024-04-13T03:51:36.988 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 342 2024-04-13T03:51:36.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.989 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-04-13T03:51:36.989 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-04-13T03:51:36.989 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:36.989 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-04-13T03:51:36.989 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-04-13T03:51:36.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.989 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-04-13T03:51:36.989 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-04-13T03:51:36.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.990 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-04-13T03:51:36.990 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:36.990 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 10, 2024-04-13T03:51:36.990 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-04-13T03:51:36.990 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 5, 2024-04-13T03:51:36.990 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 22946, 2024-04-13T03:51:36.990 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 11473, 2024-04-13T03:51:36.990 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:36.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.991 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:36.991 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:36.991 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-13T03:51:36.991 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:36.991 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 648102, 2024-04-13T03:51:36.991 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 113308687, 2024-04-13T03:51:36.991 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:36.991 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 648102, 2024-04-13T03:51:36.992 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:36.992 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:36.992 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 648102, 2024-04-13T03:51:36.992 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 113308687, 2024-04-13T03:51:36.992 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:36.992 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.992 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.992 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.993 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:36.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.993 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:36.993 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:36.993 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-13T03:51:36.993 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:36.994 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:36.994 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:36.994 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:36.994 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:36.994 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 60949, 2024-04-13T03:51:36.994 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 547180996, 2024-04-13T03:51:36.994 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 60949, 2024-04-13T03:51:36.994 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 547180996, 2024-04-13T03:51:36.994 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:36.995 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.995 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.995 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.995 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:36.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.995 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:36.995 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:36.995 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-04-13T03:51:36.996 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:36.996 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:36.996 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:36.996 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:36.996 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:36.996 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 60949, 2024-04-13T03:51:36.996 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 60949, 2024-04-13T03:51:36.996 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 60949, 2024-04-13T03:51:36.996 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 60949, 2024-04-13T03:51:36.997 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:36.997 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.997 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.997 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.997 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:36.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.997 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:36.997 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:36.997 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-13T03:51:36.998 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:36.998 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 11473, 2024-04-13T03:51:36.998 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 1252061, 2024-04-13T03:51:36.998 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:36.998 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 11473, 2024-04-13T03:51:36.998 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:36.998 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:36.998 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 565, 2024-04-13T03:51:36.998 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1252061, 2024-04-13T03:51:36.999 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:36.999 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:36.999 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:36.999 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:36.999 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:36.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:36.999 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x555b47a6c500": { 2024-04-13T03:51:36.999 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 267878, 2024-04-13T03:51:36.999 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-13T03:51:37.000 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:37.000 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 76293, 2024-04-13T03:51:37.000 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 543198437, 2024-04-13T03:51:37.000 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:37.000 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 76293, 2024-04-13T03:51:37.000 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:37.000 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:37.000 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 35077, 2024-04-13T03:51:37.000 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 542930559, 2024-04-13T03:51:37.001 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:37.001 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:37.001 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:37.001 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:37.001 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:37.001 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:37.001 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-13T03:51:37.814 INFO:tasks.check_counter:Daemon mds.b mds.exported=25 2024-04-13T03:51:37.815 INFO:tasks.check_counter:Daemon mds.b mds.imported=14 2024-04-13T03:51:37.815 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=533 2024-04-13T03:51:37.815 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=267 2024-04-13T03:51:37.815 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-13T03:51:37.815 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=17 2024-04-13T03:51:37.815 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=11 2024-04-13T03:51:37.815 DEBUG:tasks.check_counter:Getting stats from e 2024-04-13T03:51:37.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-13T03:51:38.060 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:51:38.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:38 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:38.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:38 smithi119 ceph-mon[35835]: pgmap v531: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 3.5 MiB/s wr, 153 op/s 2024-04-13T03:51:38.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:38 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:38.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:38 smithi016 ceph-mon[31772]: pgmap v531: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 3.5 MiB/s wr, 153 op/s 2024-04-13T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:38 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:38 smithi188 ceph-mon[36398]: pgmap v531: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 3.5 MiB/s wr, 153 op/s 2024-04-13T03:51:40.392 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-13T03:51:40.392 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:40.392 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 171270, 2024-04-13T03:51:40.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 133273, 2024-04-13T03:51:40.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 48581158, 2024-04-13T03:51:40.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 78879934, 2024-04-13T03:51:40.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 11, 2024-04-13T03:51:40.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-04-13T03:51:40.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 10.729101924, 2024-04-13T03:51:40.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 4.613184180, 2024-04-13T03:51:40.393 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 6.848324230, 2024-04-13T03:51:40.394 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.697445810, 2024-04-13T03:51:40.394 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:40.394 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 133272, 2024-04-13T03:51:40.394 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 6.048454853, 2024-04-13T03:51:40.394 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000045384 2024-04-13T03:51:40.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.394 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:40.394 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 280002, 2024-04-13T03:51:40.394 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.043563756, 2024-04-13T03:51:40.395 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000155 2024-04-13T03:51:40.395 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.395 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-13T03:51:40.395 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 448 2024-04-13T03:51:40.395 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.395 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:40.395 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 119011, 2024-04-13T03:51:40.395 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 152172, 2024-04-13T03:51:40.395 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 34817565, 2024-04-13T03:51:40.396 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 127536221, 2024-04-13T03:51:40.396 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 20, 2024-04-13T03:51:40.396 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 9, 2024-04-13T03:51:40.396 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 7.210405118, 2024-04-13T03:51:40.396 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 3.169492877, 2024-04-13T03:51:40.396 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 4.477312772, 2024-04-13T03:51:40.396 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.357537137, 2024-04-13T03:51:40.396 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:40.396 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 152169, 2024-04-13T03:51:40.397 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 11.369181589, 2024-04-13T03:51:40.397 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000074714 2024-04-13T03:51:40.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.397 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:40.397 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 160128, 2024-04-13T03:51:40.397 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.042936855, 2024-04-13T03:51:40.397 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000268 2024-04-13T03:51:40.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.398 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 2928, 2024-04-13T03:51:40.398 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 226800 2024-04-13T03:51:40.398 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.398 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:40.398 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 40361, 2024-04-13T03:51:40.398 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 40861, 2024-04-13T03:51:40.398 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 11806358, 2024-04-13T03:51:40.398 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 72199218, 2024-04-13T03:51:40.398 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 8, 2024-04-13T03:51:40.399 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-04-13T03:51:40.399 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 3.550727599, 2024-04-13T03:51:40.399 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 1.506210652, 2024-04-13T03:51:40.399 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 1.623885982, 2024-04-13T03:51:40.399 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.586541268, 2024-04-13T03:51:40.399 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:40.399 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 40860, 2024-04-13T03:51:40.399 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 3.053824525, 2024-04-13T03:51:40.399 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000074738 2024-04-13T03:51:40.400 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.400 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:40.400 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 58541, 2024-04-13T03:51:40.400 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.011623492, 2024-04-13T03:51:40.400 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000198 2024-04-13T03:51:40.400 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.400 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1975568, 2024-04-13T03:51:40.400 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 665312 2024-04-13T03:51:40.400 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.401 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-04-13T03:51:40.401 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-04-13T03:51:40.401 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-04-13T03:51:40.401 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.401 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-04-13T03:51:40.401 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-13T03:51:40.401 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-13T03:51:40.401 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 23637, 2024-04-13T03:51:40.401 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 30.759074091, 2024-04-13T03:51:40.402 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001301310 2024-04-13T03:51:40.402 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:40.402 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.402 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:40.402 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-13T03:51:40.402 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-13T03:51:40.402 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 14691, 2024-04-13T03:51:40.402 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.984530759, 2024-04-13T03:51:40.402 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000135084 2024-04-13T03:51:40.403 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:40.403 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.403 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-04-13T03:51:40.403 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 51718, 2024-04-13T03:51:40.403 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 51146, 2024-04-13T03:51:40.403 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-04-13T03:51:40.403 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 51146, 2024-04-13T03:51:40.403 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 192.517240408, 2024-04-13T03:51:40.403 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003764072 2024-04-13T03:51:40.404 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.404 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-04-13T03:51:40.404 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 571, 2024-04-13T03:51:40.404 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-04-13T03:51:40.404 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:40.404 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 40, 2024-04-13T03:51:40.405 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 3, 2024-04-13T03:51:40.405 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 5, 2024-04-13T03:51:40.405 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 1151, 2024-04-13T03:51:40.405 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 331, 2024-04-13T03:51:40.405 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 808, 2024-04-13T03:51:40.405 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 12, 2024-04-13T03:51:40.405 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 679, 2024-04-13T03:51:40.405 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 260, 2024-04-13T03:51:40.405 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 141, 2024-04-13T03:51:40.406 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 141, 2024-04-13T03:51:40.406 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 12, 2024-04-13T03:51:40.406 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 97404, 2024-04-13T03:51:40.406 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 54380, 2024-04-13T03:51:40.406 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 569, 2024-04-13T03:51:40.406 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 153, 2024-04-13T03:51:40.406 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:40.406 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:40.406 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 1373, 2024-04-13T03:51:40.407 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 28946, 2024-04-13T03:51:40.407 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-04-13T03:51:40.407 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 5, 2024-04-13T03:51:40.407 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 203, 2024-04-13T03:51:40.407 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 5, 2024-04-13T03:51:40.407 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 352, 2024-04-13T03:51:40.407 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:40.407 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:40.407 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:40.408 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 747, 2024-04-13T03:51:40.408 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 212049091, 2024-04-13T03:51:40.408 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 6, 2024-04-13T03:51:40.408 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:40.408 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:40.408 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:40.408 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:40.408 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:40.408 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 19, 2024-04-13T03:51:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 4970, 2024-04-13T03:51:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 29099, 2024-04-13T03:51:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 18605, 2024-04-13T03:51:40.410 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 6735, 2024-04-13T03:51:40.410 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 3528, 2024-04-13T03:51:40.410 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 39059 2024-04-13T03:51:40.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.410 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-04-13T03:51:40.410 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 308, 2024-04-13T03:51:40.410 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 2110, 2024-04-13T03:51:40.410 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 1128, 2024-04-13T03:51:40.410 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 1978, 2024-04-13T03:51:40.411 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 477, 2024-04-13T03:51:40.411 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 21, 2024-04-13T03:51:40.411 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-04-13T03:51:40.411 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:40.411 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 7176, 2024-04-13T03:51:40.411 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 7155, 2024-04-13T03:51:40.411 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:40.411 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-04-13T03:51:40.411 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:40.412 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:40.412 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:40.412 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-04-13T03:51:40.412 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-04-13T03:51:40.412 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:40.412 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:40.412 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:40.412 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 7, 2024-04-13T03:51:40.412 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-04-13T03:51:40.413 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 8, 2024-04-13T03:51:40.413 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:40.413 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-04-13T03:51:40.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.413 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-04-13T03:51:40.413 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-04-13T03:51:40.413 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 43483, 2024-04-13T03:51:40.413 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-04-13T03:51:40.413 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-04-13T03:51:40.414 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 43483, 2024-04-13T03:51:40.414 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-04-13T03:51:40.414 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-04-13T03:51:40.414 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 62, 2024-04-13T03:51:40.414 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-04-13T03:51:40.414 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-04-13T03:51:40.414 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 62, 2024-04-13T03:51:40.414 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 5, 2024-04-13T03:51:40.414 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-04-13T03:51:40.415 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-04-13T03:51:40.415 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-04-13T03:51:40.415 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 131694993, 2024-04-13T03:51:40.415 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-04-13T03:51:40.415 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-04-13T03:51:40.415 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 12148, 2024-04-13T03:51:40.415 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 70.639202971, 2024-04-13T03:51:40.415 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005814883 2024-04-13T03:51:40.415 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-04-13T03:51:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-04-13T03:51:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 604, 2024-04-13T03:51:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 7953, 2024-04-13T03:51:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 7349, 2024-04-13T03:51:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 133, 2024-04-13T03:51:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 1064, 2024-04-13T03:51:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 931, 2024-04-13T03:51:40.417 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 1115, 2024-04-13T03:51:40.417 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 14425, 2024-04-13T03:51:40.417 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 13310, 2024-04-13T03:51:40.417 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 141, 2024-04-13T03:51:40.417 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 8595, 2024-04-13T03:51:40.417 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 8454, 2024-04-13T03:51:40.417 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 71016, 2024-04-13T03:51:40.417 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 223608 2024-04-13T03:51:40.417 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.418 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-04-13T03:51:40.418 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 80465, 2024-04-13T03:51:40.418 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 1506, 2024-04-13T03:51:40.418 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 51718, 2024-04-13T03:51:40.418 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 1185, 2024-04-13T03:51:40.418 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 7893, 2024-04-13T03:51:40.418 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-04-13T03:51:40.418 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 7197, 2024-04-13T03:51:40.418 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 38.254827833, 2024-04-13T03:51:40.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005315385 2024-04-13T03:51:40.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.419 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-04-13T03:51:40.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3292, 2024-04-13T03:51:40.419 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 41.600535973, 2024-04-13T03:51:40.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.012636857 2024-04-13T03:51:40.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.419 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:40.419 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.420 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.420 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.420 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-04-13T03:51:40.420 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.420 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.420 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-04-13T03:51:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 15820, 2024-04-13T03:51:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 27.081206387, 2024-04-13T03:51:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001711833 2024-04-13T03:51:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.421 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.422 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-04-13T03:51:40.422 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.422 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.422 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.422 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-04-13T03:51:40.422 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.422 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.424 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.424 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.424 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.425 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-04-13T03:51:40.425 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.425 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.425 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.425 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.425 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-04-13T03:51:40.425 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.425 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.425 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-04-13T03:51:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-04-13T03:51:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.426 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.427 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.427 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.427 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-04-13T03:51:40.427 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.427 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.427 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.427 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.427 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-04-13T03:51:40.427 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 15651, 2024-04-13T03:51:40.428 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 45.780488466, 2024-04-13T03:51:40.428 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002925083 2024-04-13T03:51:40.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.428 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-04-13T03:51:40.428 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1862, 2024-04-13T03:51:40.428 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 10.304108417, 2024-04-13T03:51:40.428 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005533892 2024-04-13T03:51:40.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.428 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:40.429 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.429 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.429 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.429 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:40.429 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.429 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.429 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.430 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-04-13T03:51:40.430 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.430 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.430 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.430 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:40.430 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.430 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.431 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.431 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:40.431 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.431 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.431 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.431 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-04-13T03:51:40.431 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 12, 2024-04-13T03:51:40.432 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.184776256, 2024-04-13T03:51:40.432 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.015398021 2024-04-13T03:51:40.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.432 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:40.432 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.432 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.432 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.432 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:40.433 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 136, 2024-04-13T03:51:40.433 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.179785251, 2024-04-13T03:51:40.433 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001321950 2024-04-13T03:51:40.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.433 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-04-13T03:51:40.433 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.433 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.433 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.434 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-04-13T03:51:40.434 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.434 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.434 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.434 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-04-13T03:51:40.434 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.434 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.434 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.435 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-04-13T03:51:40.435 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 7176, 2024-04-13T03:51:40.435 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 29.131511825, 2024-04-13T03:51:40.435 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004059575 2024-04-13T03:51:40.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.435 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:40.435 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:40.435 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:40.436 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.436 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.436 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.436 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:40.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.436 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-04-13T03:51:40.436 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-04-13T03:51:40.436 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 3, 2024-04-13T03:51:40.436 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 2, 2024-04-13T03:51:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 1, 2024-04-13T03:51:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-04-13T03:51:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 15912, 2024-04-13T03:51:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 15912, 2024-04-13T03:51:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 877, 2024-04-13T03:51:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-04-13T03:51:40.437 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:40.438 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:40.438 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:40.438 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:40.438 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:40.438 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:40.438 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:40.438 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:40.438 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:40.438 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:40.439 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:40.439 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:40.439 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:40.439 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:40.439 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:40.439 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:40.439 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:40.439 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:40.439 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:40.440 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:40.440 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:40.440 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:40.440 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:40.440 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:40.440 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:40.440 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:40.440 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:40.440 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:40.441 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:40.441 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:40.441 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:40.441 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-04-13T03:51:40.441 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:40.441 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:40.441 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:40.441 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:40.441 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 1224816, 2024-04-13T03:51:40.442 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 836, 2024-04-13T03:51:40.442 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 440, 2024-04-13T03:51:40.442 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 5, 2024-04-13T03:51:40.442 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-04-13T03:51:40.442 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-04-13T03:51:40.442 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:40.442 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:40.442 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:40.442 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:40.443 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:40.443 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 186, 2024-04-13T03:51:40.443 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:40.443 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:40.443 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:40.443 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-04-13T03:51:40.443 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 2949364, 2024-04-13T03:51:40.443 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 39342, 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 28746, 2024-04-13T03:51:40.444 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 129419436, 2024-04-13T03:51:40.445 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 33, 2024-04-13T03:51:40.445 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 28746, 2024-04-13T03:51:40.445 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-04-13T03:51:40.445 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 28746, 2024-04-13T03:51:40.445 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 262.981577589, 2024-04-13T03:51:40.445 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.009148458 2024-04-13T03:51:40.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.445 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-04-13T03:51:40.445 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-04-13T03:51:40.446 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 28746, 2024-04-13T03:51:40.446 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 29711 2024-04-13T03:51:40.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.446 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 28746, 2024-04-13T03:51:40.446 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 598, 2024-04-13T03:51:40.446 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 28148, 2024-04-13T03:51:40.446 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-04-13T03:51:40.446 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-04-13T03:51:40.447 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 0, 2024-04-13T03:51:40.447 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 10, 2024-04-13T03:51:40.447 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 598, 2024-04-13T03:51:40.447 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 12746, 2024-04-13T03:51:40.447 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 362, 2024-04-13T03:51:40.447 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-04-13T03:51:40.447 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-04-13T03:51:40.447 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-04-13T03:51:40.447 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 3, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 14378, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 0, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 20, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:40.448 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-04-13T03:51:40.449 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-04-13T03:51:40.449 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-04-13T03:51:40.449 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:40.449 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-04-13T03:51:40.449 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:40.449 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-04-13T03:51:40.449 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-04-13T03:51:40.449 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-04-13T03:51:40.449 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-04-13T03:51:40.450 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-04-13T03:51:40.450 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-04-13T03:51:40.450 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-04-13T03:51:40.450 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-04-13T03:51:40.450 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-04-13T03:51:40.450 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-04-13T03:51:40.450 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-04-13T03:51:40.450 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-04-13T03:51:40.450 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-04-13T03:51:40.451 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-04-13T03:51:40.451 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-04-13T03:51:40.451 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-04-13T03:51:40.451 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-04-13T03:51:40.451 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 117, 2024-04-13T03:51:40.451 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-04-13T03:51:40.451 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 27, 2024-04-13T03:51:40.451 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-04-13T03:51:40.451 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-04-13T03:51:40.452 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-04-13T03:51:40.452 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-04-13T03:51:40.452 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 1226, 2024-04-13T03:51:40.452 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 0, 2024-04-13T03:51:40.452 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 368 2024-04-13T03:51:40.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.452 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-04-13T03:51:40.452 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-04-13T03:51:40.452 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:40.453 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-04-13T03:51:40.453 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-04-13T03:51:40.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.453 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-04-13T03:51:40.453 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-04-13T03:51:40.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.453 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-04-13T03:51:40.453 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:40.453 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 10, 2024-04-13T03:51:40.454 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-04-13T03:51:40.454 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 5, 2024-04-13T03:51:40.454 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 14310, 2024-04-13T03:51:40.454 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 7155, 2024-04-13T03:51:40.454 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:40.454 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.454 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:40.454 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-13T03:51:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 330638, 2024-04-13T03:51:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 66653635, 2024-04-13T03:51:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 330638, 2024-04-13T03:51:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 330638, 2024-04-13T03:51:40.456 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 66653635, 2024-04-13T03:51:40.456 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:40.456 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.456 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.456 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.456 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:40.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.456 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:40.456 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:40.457 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-13T03:51:40.457 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:40.457 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:40.457 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:40.457 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:40.457 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:40.457 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 28746, 2024-04-13T03:51:40.457 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 130200411, 2024-04-13T03:51:40.457 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 28746, 2024-04-13T03:51:40.458 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 130200411, 2024-04-13T03:51:40.458 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:40.458 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.458 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.458 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.458 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:40.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.458 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:40.458 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:40.459 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-04-13T03:51:40.459 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:40.459 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:40.459 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:40.459 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:40.459 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:40.459 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 28746, 2024-04-13T03:51:40.459 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 28746, 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 28746, 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 28746, 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:40.460 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:40.461 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-13T03:51:40.461 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:40.461 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 7155, 2024-04-13T03:51:40.461 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 780975, 2024-04-13T03:51:40.461 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:40.461 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 7155, 2024-04-13T03:51:40.461 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:40.461 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:40.462 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 598, 2024-04-13T03:51:40.462 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 780975, 2024-04-13T03:51:40.462 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:40.462 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.462 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.462 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.462 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:40.462 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:40.462 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x558fe4b8c500": { 2024-04-13T03:51:40.463 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 19306, 2024-04-13T03:51:40.463 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-13T03:51:40.463 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:40.463 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 43483, 2024-04-13T03:51:40.463 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 127500689, 2024-04-13T03:51:40.463 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:40.463 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 43483, 2024-04-13T03:51:40.463 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:40.463 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:40.464 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 12148, 2024-04-13T03:51:40.464 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 127481383, 2024-04-13T03:51:40.464 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:40.464 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:40.464 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:40.464 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:40.464 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:40.464 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:40.464 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-13T03:51:41.223 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:40 smithi119 ceph-mon[35835]: pgmap v532: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-04-13T03:51:41.224 INFO:tasks.check_counter:Daemon mds.e mds.exported=5 2024-04-13T03:51:41.224 INFO:tasks.check_counter:Daemon mds.e mds.imported=5 2024-04-13T03:51:41.224 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=308 2024-04-13T03:51:41.225 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=2110 2024-04-13T03:51:41.225 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-04-13T03:51:41.225 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-13T03:51:41.225 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-13T03:51:41.225 DEBUG:tasks.check_counter:Getting stats from h 2024-04-13T03:51:41.225 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-13T03:51:41.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:40 smithi016 ceph-mon[31772]: pgmap v532: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-04-13T03:51:41.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:40 smithi188 ceph-mon[36398]: pgmap v532: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-04-13T03:51:41.471 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:51:43.078 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:42 smithi119 ceph-mon[35835]: pgmap v533: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 3.6 MiB/s wr, 150 op/s 2024-04-13T03:51:43.079 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-13T03:51:43.079 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:43.079 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 154036, 2024-04-13T03:51:43.079 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 209519, 2024-04-13T03:51:43.079 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 47487123, 2024-04-13T03:51:43.080 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 154425362, 2024-04-13T03:51:43.080 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 18, 2024-04-13T03:51:43.080 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 8, 2024-04-13T03:51:43.080 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 9.307998206, 2024-04-13T03:51:43.080 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 4.055248926, 2024-04-13T03:51:43.080 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 5.758716922, 2024-04-13T03:51:43.080 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.597011044, 2024-04-13T03:51:43.080 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:43.080 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 209515, 2024-04-13T03:51:43.081 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 13.542397861, 2024-04-13T03:51:43.081 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000064636 2024-04-13T03:51:43.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.081 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:43.081 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 207983, 2024-04-13T03:51:43.081 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.060056428, 2024-04-13T03:51:43.081 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000288 2024-04-13T03:51:43.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.081 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1992352, 2024-04-13T03:51:43.082 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 216624 2024-04-13T03:51:43.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.082 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:43.082 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 203313, 2024-04-13T03:51:43.082 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 171111, 2024-04-13T03:51:43.082 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 56576644, 2024-04-13T03:51:43.082 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 83453270, 2024-04-13T03:51:43.082 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 12, 2024-04-13T03:51:43.083 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 6, 2024-04-13T03:51:43.083 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 12.747800845, 2024-04-13T03:51:43.083 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 5.793279946, 2024-04-13T03:51:43.083 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 8.135860303, 2024-04-13T03:51:43.083 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.550723264, 2024-04-13T03:51:43.083 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:43.083 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 171115, 2024-04-13T03:51:43.083 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 7.331959659, 2024-04-13T03:51:43.083 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000042848 2024-04-13T03:51:43.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.084 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:43.084 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 350568, 2024-04-13T03:51:43.084 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.056273860, 2024-04-13T03:51:43.084 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000160 2024-04-13T03:51:43.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.084 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 4272, 2024-04-13T03:51:43.084 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 223552 2024-04-13T03:51:43.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 17264, 2024-04-13T03:51:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 17530, 2024-04-13T03:51:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 5517434, 2024-04-13T03:51:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 86374594, 2024-04-13T03:51:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 11, 2024-04-13T03:51:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 9, 2024-04-13T03:51:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 1.759960047, 2024-04-13T03:51:43.085 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.605674500, 2024-04-13T03:51:43.086 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.303768970, 2024-04-13T03:51:43.086 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.748949876, 2024-04-13T03:51:43.086 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:43.086 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 17524, 2024-04-13T03:51:43.086 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.327840648, 2024-04-13T03:51:43.086 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000075772 2024-04-13T03:51:43.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.086 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:43.086 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 75, 2024-04-13T03:51:43.087 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000017608, 2024-04-13T03:51:43.087 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000234 2024-04-13T03:51:43.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.087 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 4448, 2024-04-13T03:51:43.087 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 473088 2024-04-13T03:51:43.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.087 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-04-13T03:51:43.087 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-04-13T03:51:43.087 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-04-13T03:51:43.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.088 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-04-13T03:51:43.088 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-13T03:51:43.088 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-13T03:51:43.088 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 27871, 2024-04-13T03:51:43.088 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 35.716151092, 2024-04-13T03:51:43.088 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001281480 2024-04-13T03:51:43.088 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:43.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.089 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:43.089 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-13T03:51:43.089 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-13T03:51:43.089 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 21267, 2024-04-13T03:51:43.089 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 2.290755678, 2024-04-13T03:51:43.089 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000107714 2024-04-13T03:51:43.089 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:43.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.089 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-04-13T03:51:43.090 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 78751, 2024-04-13T03:51:43.090 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 65141, 2024-04-13T03:51:43.090 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-04-13T03:51:43.090 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 65141, 2024-04-13T03:51:43.090 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 234.654829908, 2024-04-13T03:51:43.090 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003602260 2024-04-13T03:51:43.090 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.090 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-04-13T03:51:43.091 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 13609, 2024-04-13T03:51:43.091 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-04-13T03:51:43.091 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:43.091 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 22, 2024-04-13T03:51:43.091 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 1, 2024-04-13T03:51:43.091 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 3, 2024-04-13T03:51:43.091 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 938, 2024-04-13T03:51:43.091 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 161, 2024-04-13T03:51:43.092 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 763, 2024-04-13T03:51:43.092 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 14, 2024-04-13T03:51:43.092 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 695, 2024-04-13T03:51:43.092 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 364, 2024-04-13T03:51:43.092 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 164, 2024-04-13T03:51:43.092 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 164, 2024-04-13T03:51:43.092 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 14, 2024-04-13T03:51:43.092 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 136500, 2024-04-13T03:51:43.093 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 60523, 2024-04-13T03:51:43.093 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 13602, 2024-04-13T03:51:43.093 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 196, 2024-04-13T03:51:43.093 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:43.093 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:43.093 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 1007, 2024-04-13T03:51:43.093 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 23414, 2024-04-13T03:51:43.093 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-04-13T03:51:43.093 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 6, 2024-04-13T03:51:43.094 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 185, 2024-04-13T03:51:43.094 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 9, 2024-04-13T03:51:43.094 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 430, 2024-04-13T03:51:43.094 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:43.094 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:43.094 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:43.094 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 747, 2024-04-13T03:51:43.094 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 212049091, 2024-04-13T03:51:43.095 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 6, 2024-04-13T03:51:43.095 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:43.095 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:43.095 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:43.095 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:43.095 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:43.095 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:43.095 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:43.095 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:43.096 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 16, 2024-04-13T03:51:43.096 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 9010, 2024-04-13T03:51:43.096 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 41429, 2024-04-13T03:51:43.096 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:43.096 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:43.096 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:43.096 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 19843, 2024-04-13T03:51:43.096 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 5787, 2024-04-13T03:51:43.096 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 2503, 2024-04-13T03:51:43.097 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 53134 2024-04-13T03:51:43.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.097 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-04-13T03:51:43.097 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 515, 2024-04-13T03:51:43.097 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 1812, 2024-04-13T03:51:43.097 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 979, 2024-04-13T03:51:43.098 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 1735, 2024-04-13T03:51:43.098 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 703, 2024-04-13T03:51:43.098 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 29, 2024-04-13T03:51:43.098 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-04-13T03:51:43.098 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:43.098 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 10664, 2024-04-13T03:51:43.098 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 10635, 2024-04-13T03:51:43.098 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:43.098 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-04-13T03:51:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-04-13T03:51:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-04-13T03:51:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 8, 2024-04-13T03:51:43.100 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-04-13T03:51:43.100 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 4, 2024-04-13T03:51:43.100 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:43.100 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-04-13T03:51:43.100 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.100 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-04-13T03:51:43.100 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-04-13T03:51:43.100 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 50906, 2024-04-13T03:51:43.100 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-04-13T03:51:43.101 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-04-13T03:51:43.101 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 50906, 2024-04-13T03:51:43.101 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-04-13T03:51:43.101 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-04-13T03:51:43.101 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 71, 2024-04-13T03:51:43.101 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-04-13T03:51:43.101 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-04-13T03:51:43.101 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 71, 2024-04-13T03:51:43.101 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 6, 2024-04-13T03:51:43.102 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-04-13T03:51:43.102 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-04-13T03:51:43.102 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-04-13T03:51:43.102 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 153921392, 2024-04-13T03:51:43.102 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-04-13T03:51:43.102 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-04-13T03:51:43.102 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 14305, 2024-04-13T03:51:43.102 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 79.206475185, 2024-04-13T03:51:43.103 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005536978 2024-04-13T03:51:43.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.103 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-04-13T03:51:43.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.103 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-04-13T03:51:43.103 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 641, 2024-04-13T03:51:43.103 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 11533, 2024-04-13T03:51:43.103 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 10892, 2024-04-13T03:51:43.103 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 129, 2024-04-13T03:51:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 933, 2024-04-13T03:51:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 804, 2024-04-13T03:51:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 934, 2024-04-13T03:51:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 19231, 2024-04-13T03:51:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 18297, 2024-04-13T03:51:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 135, 2024-04-13T03:51:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 11628, 2024-04-13T03:51:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 11493, 2024-04-13T03:51:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 70228, 2024-04-13T03:51:43.105 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 223608 2024-04-13T03:51:43.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.105 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-04-13T03:51:43.105 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 119729, 2024-04-13T03:51:43.105 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 984, 2024-04-13T03:51:43.105 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 78751, 2024-04-13T03:51:43.105 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 982, 2024-04-13T03:51:43.105 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 11721, 2024-04-13T03:51:43.105 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-04-13T03:51:43.106 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 10664, 2024-04-13T03:51:43.106 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 52.748039891, 2024-04-13T03:51:43.106 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004946365 2024-04-13T03:51:43.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.106 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-04-13T03:51:43.106 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2593, 2024-04-13T03:51:43.106 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 34.101147551, 2024-04-13T03:51:43.106 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.013151233 2024-04-13T03:51:43.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.107 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:43.107 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.107 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.107 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.107 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-04-13T03:51:43.107 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.107 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.107 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.108 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-04-13T03:51:43.108 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 18014, 2024-04-13T03:51:43.108 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 51.410374152, 2024-04-13T03:51:43.108 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002853912 2024-04-13T03:51:43.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.108 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:43.108 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.108 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.109 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.109 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-04-13T03:51:43.109 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.109 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.109 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.109 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-04-13T03:51:43.109 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.110 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.110 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.110 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:43.110 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.110 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.110 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.111 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:43.111 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.111 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.111 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.111 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-04-13T03:51:43.111 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.111 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.111 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.112 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-04-13T03:51:43.112 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.112 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.112 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.112 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-04-13T03:51:43.112 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.113 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.113 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.113 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-04-13T03:51:43.113 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.113 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.113 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.113 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-04-13T03:51:43.114 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.114 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.114 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.114 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-04-13T03:51:43.114 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 21570, 2024-04-13T03:51:43.114 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 50.331706801, 2024-04-13T03:51:43.114 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002333412 2024-04-13T03:51:43.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.115 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-04-13T03:51:43.115 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1379, 2024-04-13T03:51:43.115 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 11.104946351, 2024-04-13T03:51:43.115 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.008052898 2024-04-13T03:51:43.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.115 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:43.115 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.115 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.116 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.116 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:43.116 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.116 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.116 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.116 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-04-13T03:51:43.116 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.117 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.117 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.117 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:43.117 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.117 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.117 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.117 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:43.118 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.118 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.118 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.118 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-04-13T03:51:43.118 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 13, 2024-04-13T03:51:43.118 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.426334225, 2024-04-13T03:51:43.118 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.032794940 2024-04-13T03:51:43.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.119 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:43.119 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.119 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.119 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.119 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:43.119 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 244, 2024-04-13T03:51:43.119 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.396790734, 2024-04-13T03:51:43.120 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001626191 2024-04-13T03:51:43.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.120 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-04-13T03:51:43.120 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.120 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.120 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.121 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-04-13T03:51:43.121 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.121 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.121 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.121 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-04-13T03:51:43.121 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.121 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.122 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.122 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-04-13T03:51:43.122 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 10664, 2024-04-13T03:51:43.122 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 34.135490203, 2024-04-13T03:51:43.122 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003201002 2024-04-13T03:51:43.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.122 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:43.122 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:43.123 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:43.123 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.123 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.123 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.123 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:43.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.123 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-04-13T03:51:43.123 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 2, 2024-04-13T03:51:43.123 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 5, 2024-04-13T03:51:43.124 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 3, 2024-04-13T03:51:43.124 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 2, 2024-04-13T03:51:43.124 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-04-13T03:51:43.124 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 18022, 2024-04-13T03:51:43.124 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 9011, 2024-04-13T03:51:43.124 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 961, 2024-04-13T03:51:43.124 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:43.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.124 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-04-13T03:51:43.125 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:43.125 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:43.125 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:43.125 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:43.125 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:43.125 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:43.125 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:43.125 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:43.125 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:43.126 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:43.126 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:43.126 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:43.126 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:43.126 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:43.126 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:43.126 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:43.126 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:43.126 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:43.127 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:43.127 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:43.127 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:43.127 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:43.127 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:43.127 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:43.127 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:43.127 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:43.127 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:43.128 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:43.128 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:43.128 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:43.128 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:43.128 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-04-13T03:51:43.128 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:43.128 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:43.128 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:43.128 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:43.129 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 620708, 2024-04-13T03:51:43.129 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 703, 2024-04-13T03:51:43.129 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-04-13T03:51:43.129 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-04-13T03:51:43.129 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-04-13T03:51:43.129 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-04-13T03:51:43.129 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:43.129 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:43.129 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:43.130 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:43.130 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:43.130 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 186, 2024-04-13T03:51:43.130 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:43.130 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:43.130 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:43.130 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-04-13T03:51:43.130 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 2767404, 2024-04-13T03:51:43.130 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 32112, 2024-04-13T03:51:43.131 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:43.131 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-04-13T03:51:43.131 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:43.131 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-04-13T03:51:43.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.131 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-04-13T03:51:43.131 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-04-13T03:51:43.131 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-04-13T03:51:43.131 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 37801, 2024-04-13T03:51:43.132 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 151745506, 2024-04-13T03:51:43.132 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 33, 2024-04-13T03:51:43.132 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 37801, 2024-04-13T03:51:43.132 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-04-13T03:51:43.132 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 37801, 2024-04-13T03:51:43.132 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 296.871741758, 2024-04-13T03:51:43.132 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.007853542 2024-04-13T03:51:43.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.132 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-04-13T03:51:43.133 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-04-13T03:51:43.133 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 37801, 2024-04-13T03:51:43.133 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 38717 2024-04-13T03:51:43.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.133 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 37801, 2024-04-13T03:51:43.133 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 591, 2024-04-13T03:51:43.133 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 37210, 2024-04-13T03:51:43.133 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-04-13T03:51:43.133 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-04-13T03:51:43.134 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 0, 2024-04-13T03:51:43.134 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 10, 2024-04-13T03:51:43.134 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 591, 2024-04-13T03:51:43.134 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 14896, 2024-04-13T03:51:43.134 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 376, 2024-04-13T03:51:43.134 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-04-13T03:51:43.134 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-04-13T03:51:43.134 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-04-13T03:51:43.135 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 1, 2024-04-13T03:51:43.135 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 21327, 2024-04-13T03:51:43.135 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-04-13T03:51:43.135 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:43.135 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:43.135 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 0, 2024-04-13T03:51:43.135 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 20, 2024-04-13T03:51:43.135 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:43.135 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:43.136 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:43.136 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-04-13T03:51:43.136 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-04-13T03:51:43.136 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-04-13T03:51:43.136 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:43.136 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-04-13T03:51:43.136 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:43.136 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-04-13T03:51:43.136 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-04-13T03:51:43.137 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-04-13T03:51:43.137 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-04-13T03:51:43.137 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-04-13T03:51:43.137 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-04-13T03:51:43.137 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-04-13T03:51:43.137 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-04-13T03:51:43.137 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-04-13T03:51:43.137 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-04-13T03:51:43.137 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-04-13T03:51:43.138 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-04-13T03:51:43.138 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-04-13T03:51:43.138 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-04-13T03:51:43.138 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-04-13T03:51:43.138 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-04-13T03:51:43.138 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-04-13T03:51:43.138 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 117, 2024-04-13T03:51:43.138 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-04-13T03:51:43.139 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 27, 2024-04-13T03:51:43.139 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-04-13T03:51:43.139 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-04-13T03:51:43.139 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-04-13T03:51:43.139 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-04-13T03:51:43.139 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 1084, 2024-04-13T03:51:43.139 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 0, 2024-04-13T03:51:43.139 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 412 2024-04-13T03:51:43.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.140 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-04-13T03:51:43.140 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-04-13T03:51:43.140 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:43.140 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-04-13T03:51:43.140 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-04-13T03:51:43.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.140 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-04-13T03:51:43.140 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 2 2024-04-13T03:51:43.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.141 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-04-13T03:51:43.141 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:43.141 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 10, 2024-04-13T03:51:43.141 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-04-13T03:51:43.141 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 5, 2024-04-13T03:51:43.141 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 21270, 2024-04-13T03:51:43.141 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 10635, 2024-04-13T03:51:43.141 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:43.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.142 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:43.142 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:43.142 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-13T03:51:43.142 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:43.142 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 374609, 2024-04-13T03:51:43.142 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 77543193, 2024-04-13T03:51:43.142 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:43.142 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 374609, 2024-04-13T03:51:43.143 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:43.143 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:43.143 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 374609, 2024-04-13T03:51:43.143 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 77543193, 2024-04-13T03:51:43.143 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:43.143 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.143 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.143 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.144 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:43.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.144 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:43.144 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:43.145 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-13T03:51:43.145 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:43.145 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:43.145 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:43.145 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:43.145 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:43.145 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 37801, 2024-04-13T03:51:43.145 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 152905953, 2024-04-13T03:51:43.146 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 37801, 2024-04-13T03:51:43.146 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 152905953, 2024-04-13T03:51:43.146 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:43.146 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.146 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.146 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.146 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:43.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.146 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:43.147 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:43.147 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-04-13T03:51:43.147 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:43.147 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:43.147 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:43.147 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:43.147 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:43.147 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 37801, 2024-04-13T03:51:43.147 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 37801, 2024-04-13T03:51:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 37801, 2024-04-13T03:51:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 37801, 2024-04-13T03:51:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.148 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:43.149 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:43.149 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-13T03:51:43.149 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:43.149 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 10635, 2024-04-13T03:51:43.149 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 1160447, 2024-04-13T03:51:43.149 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:43.149 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 10635, 2024-04-13T03:51:43.149 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:43.149 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:43.150 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 591, 2024-04-13T03:51:43.150 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1160447, 2024-04-13T03:51:43.150 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:43.150 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.150 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.150 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.150 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:43.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:43.151 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x558bc9cde500": { 2024-04-13T03:51:43.151 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 326216, 2024-04-13T03:51:43.151 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-13T03:51:43.151 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:43.151 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 50906, 2024-04-13T03:51:43.151 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 149727088, 2024-04-13T03:51:43.151 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:43.151 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 50906, 2024-04-13T03:51:43.151 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:43.152 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:43.152 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 14305, 2024-04-13T03:51:43.152 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 149400872, 2024-04-13T03:51:43.152 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:43.152 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:43.152 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:43.152 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:43.152 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:43.152 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:43.153 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-13T03:51:43.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:42 smithi016 ceph-mon[31772]: pgmap v533: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 3.6 MiB/s wr, 150 op/s 2024-04-13T03:51:43.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:42 smithi188 ceph-mon[36398]: pgmap v533: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 3.6 MiB/s wr, 150 op/s 2024-04-13T03:51:44.394 INFO:tasks.check_counter:Daemon mds.h mds.exported=6 2024-04-13T03:51:44.395 INFO:tasks.check_counter:Daemon mds.h mds.imported=9 2024-04-13T03:51:44.395 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=515 2024-04-13T03:51:44.395 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=1812 2024-04-13T03:51:44.395 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-13T03:51:44.395 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-13T03:51:44.395 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-13T03:51:44.395 DEBUG:tasks.check_counter:Getting stats from k 2024-04-13T03:51:44.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-13T03:51:44.639 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.b/config 2024-04-13T03:51:45.031 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:44 smithi119 ceph-mon[35835]: pgmap v534: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-04-13T03:51:45.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:44 smithi016 ceph-mon[31772]: pgmap v534: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-04-13T03:51:45.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:44 smithi188 ceph-mon[36398]: pgmap v534: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-04-13T03:51:46.400 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-13T03:51:46.400 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:46.401 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 2179, 2024-04-13T03:51:46.401 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 2179, 2024-04-13T03:51:46.401 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 42596785, 2024-04-13T03:51:46.401 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 692099, 2024-04-13T03:51:46.401 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 8, 2024-04-13T03:51:46.401 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 4, 2024-04-13T03:51:46.401 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.195111466, 2024-04-13T03:51:46.402 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.058943739, 2024-04-13T03:51:46.402 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.067402475, 2024-04-13T03:51:46.402 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.052305188, 2024-04-13T03:51:46.402 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:46.402 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2178, 2024-04-13T03:51:46.402 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.077610034, 2024-04-13T03:51:46.403 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000035633 2024-04-13T03:51:46.403 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.403 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:46.403 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.403 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.403 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.403 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.404 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-13T03:51:46.404 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 448 2024-04-13T03:51:46.404 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.404 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:46.404 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1786, 2024-04-13T03:51:46.404 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1730, 2024-04-13T03:51:46.404 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 69422904, 2024-04-13T03:51:46.405 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 645400, 2024-04-13T03:51:46.405 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 5, 2024-04-13T03:51:46.405 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 4, 2024-04-13T03:51:46.405 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.196820185, 2024-04-13T03:51:46.405 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.041912939, 2024-04-13T03:51:46.405 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.114988014, 2024-04-13T03:51:46.405 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.041577727, 2024-04-13T03:51:46.405 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:46.406 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1729, 2024-04-13T03:51:46.406 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.054031866, 2024-04-13T03:51:46.406 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000031250 2024-04-13T03:51:46.406 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.407 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:46.407 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.407 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.407 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.407 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.407 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1967872, 2024-04-13T03:51:46.408 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 210944 2024-04-13T03:51:46.408 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.408 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:46.408 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 506, 2024-04-13T03:51:46.408 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 758, 2024-04-13T03:51:46.408 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 16376757, 2024-04-13T03:51:46.409 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 923762, 2024-04-13T03:51:46.409 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 10, 2024-04-13T03:51:46.409 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 6, 2024-04-13T03:51:46.409 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.059500783, 2024-04-13T03:51:46.409 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.018901662, 2024-04-13T03:51:46.410 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.022727671, 2024-04-13T03:51:46.410 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.012655800, 2024-04-13T03:51:46.410 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:46.410 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 755, 2024-04-13T03:51:46.410 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.043771050, 2024-04-13T03:51:46.410 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000057974 2024-04-13T03:51:46.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.411 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:46.411 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.411 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.411 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.411 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-13T03:51:46.412 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 763808 2024-04-13T03:51:46.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.412 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-04-13T03:51:46.412 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-04-13T03:51:46.412 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-04-13T03:51:46.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.412 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-04-13T03:51:46.413 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-13T03:51:46.413 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-13T03:51:46.413 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 6881, 2024-04-13T03:51:46.413 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.147701527, 2024-04-13T03:51:46.413 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000021465 2024-04-13T03:51:46.413 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:46.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.413 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:46.413 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-13T03:51:46.414 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-13T03:51:46.414 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.414 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.414 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.414 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:46.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.414 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-04-13T03:51:46.414 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 0, 2024-04-13T03:51:46.415 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 0, 2024-04-13T03:51:46.415 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-04-13T03:51:46.415 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.415 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.415 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.415 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.415 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-04-13T03:51:46.415 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 0, 2024-04-13T03:51:46.416 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-04-13T03:51:46.416 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:46.416 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 0, 2024-04-13T03:51:46.416 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 0, 2024-04-13T03:51:46.417 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-04-13T03:51:46.417 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 8485, 2024-04-13T03:51:46.417 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 17, 2024-04-13T03:51:46.417 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 8468, 2024-04-13T03:51:46.417 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 0, 2024-04-13T03:51:46.418 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 8461, 2024-04-13T03:51:46.418 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 0, 2024-04-13T03:51:46.418 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 0, 2024-04-13T03:51:46.418 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 0, 2024-04-13T03:51:46.418 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 10, 2024-04-13T03:51:46.418 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 0, 2024-04-13T03:51:46.418 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 0, 2024-04-13T03:51:46.418 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 0, 2024-04-13T03:51:46.419 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 0, 2024-04-13T03:51:46.419 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:46.419 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:46.419 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 0, 2024-04-13T03:51:46.419 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-04-13T03:51:46.419 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-04-13T03:51:46.419 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 0, 2024-04-13T03:51:46.419 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 0, 2024-04-13T03:51:46.420 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 0, 2024-04-13T03:51:46.420 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 0, 2024-04-13T03:51:46.420 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:46.420 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:46.420 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:46.420 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 0, 2024-04-13T03:51:46.420 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 0, 2024-04-13T03:51:46.421 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-04-13T03:51:46.421 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:46.421 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:46.421 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:46.421 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:46.421 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:46.421 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:46.421 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:46.422 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:46.422 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 0, 2024-04-13T03:51:46.422 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 0, 2024-04-13T03:51:46.422 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 0, 2024-04-13T03:51:46.422 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:46.422 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:46.422 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:46.422 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 0, 2024-04-13T03:51:46.422 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 0, 2024-04-13T03:51:46.423 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 0, 2024-04-13T03:51:46.423 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 0 2024-04-13T03:51:46.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.423 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-04-13T03:51:46.423 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 0, 2024-04-13T03:51:46.423 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 0, 2024-04-13T03:51:46.423 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 0, 2024-04-13T03:51:46.423 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 0, 2024-04-13T03:51:46.423 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 0, 2024-04-13T03:51:46.424 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-04-13T03:51:46.424 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-04-13T03:51:46.424 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:46.424 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 0, 2024-04-13T03:51:46.424 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 0, 2024-04-13T03:51:46.424 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:46.424 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-04-13T03:51:46.424 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:46.424 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:46.425 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:46.425 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-04-13T03:51:46.425 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-04-13T03:51:46.425 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:46.425 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:46.425 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:46.425 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 0, 2024-04-13T03:51:46.425 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-04-13T03:51:46.425 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 0, 2024-04-13T03:51:46.426 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:46.426 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-04-13T03:51:46.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.426 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-04-13T03:51:46.426 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-04-13T03:51:46.426 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 0, 2024-04-13T03:51:46.426 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-04-13T03:51:46.426 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-04-13T03:51:46.426 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 52423, 2024-04-13T03:51:46.427 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-04-13T03:51:46.427 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-04-13T03:51:46.427 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 0, 2024-04-13T03:51:46.427 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-04-13T03:51:46.427 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-04-13T03:51:46.427 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 73, 2024-04-13T03:51:46.427 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 6, 2024-04-13T03:51:46.427 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-04-13T03:51:46.427 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-04-13T03:51:46.428 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-04-13T03:51:46.428 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 129495685, 2024-04-13T03:51:46.428 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 129491014, 2024-04-13T03:51:46.428 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-04-13T03:51:46.428 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.428 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.428 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 52423 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 597, 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 8558, 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 7961, 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 45, 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 61, 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 16, 2024-04-13T03:51:46.429 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 8485, 2024-04-13T03:51:46.430 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 8707, 2024-04-13T03:51:46.430 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 222, 2024-04-13T03:51:46.430 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 0, 2024-04-13T03:51:46.430 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 0, 2024-04-13T03:51:46.430 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 0, 2024-04-13T03:51:46.430 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 61120, 2024-04-13T03:51:46.430 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 215416 2024-04-13T03:51:46.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.431 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-04-13T03:51:46.431 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 0, 2024-04-13T03:51:46.431 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 0, 2024-04-13T03:51:46.431 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 0, 2024-04-13T03:51:46.431 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 0, 2024-04-13T03:51:46.431 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 0, 2024-04-13T03:51:46.431 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-04-13T03:51:46.431 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.431 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.432 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.432 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-04-13T03:51:46.432 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.432 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.432 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.432 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:46.432 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.433 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.433 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.433 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-04-13T03:51:46.433 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.433 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.433 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.433 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-04-13T03:51:46.434 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.434 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.434 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.434 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:46.434 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.434 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.434 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.435 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-04-13T03:51:46.435 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.435 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.435 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.435 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-04-13T03:51:46.435 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.435 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.436 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.436 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:46.436 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.436 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.436 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.436 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:46.436 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.437 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.437 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.437 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-04-13T03:51:46.437 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.437 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.437 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.438 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-04-13T03:51:46.438 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.438 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.439 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.439 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-04-13T03:51:46.439 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.439 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.439 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.439 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-04-13T03:51:46.439 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.440 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.440 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.440 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-04-13T03:51:46.440 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.440 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.440 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.440 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-04-13T03:51:46.441 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.441 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.441 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.441 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-04-13T03:51:46.441 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.441 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.441 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.442 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:46.442 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.442 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.442 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.442 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.442 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:46.442 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.442 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.442 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.443 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-04-13T03:51:46.443 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.443 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.443 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.443 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:46.443 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.443 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.444 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.444 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:46.444 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.444 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.444 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.444 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-04-13T03:51:46.444 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.445 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.445 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.445 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:46.445 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.445 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.445 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.445 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:46.446 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.446 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.446 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.446 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-04-13T03:51:46.446 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.446 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.446 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.447 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-04-13T03:51:46.447 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.447 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.447 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.447 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-04-13T03:51:46.447 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.447 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.448 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.448 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-04-13T03:51:46.448 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.448 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.448 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.448 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:46.448 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:46.449 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:46.449 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.449 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.449 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.449 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:46.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.449 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-04-13T03:51:46.449 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 2, 2024-04-13T03:51:46.449 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 3, 2024-04-13T03:51:46.450 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 1, 2024-04-13T03:51:46.450 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 2, 2024-04-13T03:51:46.450 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-04-13T03:51:46.450 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 0, 2024-04-13T03:51:46.450 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 0, 2024-04-13T03:51:46.450 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 4252, 2024-04-13T03:51:46.450 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:46.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.450 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-04-13T03:51:46.451 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:46.451 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:46.451 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:46.451 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:46.451 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:46.451 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:46.451 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:46.451 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:46.451 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:46.452 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:46.452 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:46.452 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:46.452 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:46.452 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:46.452 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:46.452 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:46.452 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:46.452 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:46.453 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:46.453 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:46.453 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:46.453 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:46.453 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:46.453 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:46.453 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:46.453 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:46.453 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:46.454 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:46.454 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:46.454 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:46.454 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:46.454 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-04-13T03:51:46.454 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:46.454 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:46.454 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:46.455 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:46.455 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 256320, 2024-04-13T03:51:46.455 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 731, 2024-04-13T03:51:46.455 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-04-13T03:51:46.455 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-04-13T03:51:46.455 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-04-13T03:51:46.455 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-04-13T03:51:46.455 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:46.455 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:46.456 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:46.456 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:46.456 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:46.456 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 186, 2024-04-13T03:51:46.456 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:46.456 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:46.456 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:46.456 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-04-13T03:51:46.456 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 7653681, 2024-04-13T03:51:46.457 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 55778, 2024-04-13T03:51:46.457 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:46.457 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-04-13T03:51:46.457 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:46.457 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-04-13T03:51:46.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.457 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-04-13T03:51:46.457 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-04-13T03:51:46.457 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-04-13T03:51:46.458 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 4044, 2024-04-13T03:51:46.458 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 21, 2024-04-13T03:51:46.458 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 3, 2024-04-13T03:51:46.458 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 4044, 2024-04-13T03:51:46.458 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-04-13T03:51:46.458 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 4044, 2024-04-13T03:51:46.458 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 3.999901622, 2024-04-13T03:51:46.458 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000989095 2024-04-13T03:51:46.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.459 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-04-13T03:51:46.459 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-04-13T03:51:46.459 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 4044, 2024-04-13T03:51:46.459 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 4045 2024-04-13T03:51:46.459 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.459 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 4044, 2024-04-13T03:51:46.459 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 4044, 2024-04-13T03:51:46.459 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 0, 2024-04-13T03:51:46.459 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-04-13T03:51:46.460 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-04-13T03:51:46.460 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 2242, 2024-04-13T03:51:46.460 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 0, 2024-04-13T03:51:46.460 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 1800, 2024-04-13T03:51:46.460 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 0, 2024-04-13T03:51:46.460 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 0, 2024-04-13T03:51:46.460 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-04-13T03:51:46.460 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-04-13T03:51:46.460 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-04-13T03:51:46.461 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 0, 2024-04-13T03:51:46.461 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 0, 2024-04-13T03:51:46.461 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-04-13T03:51:46.461 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:46.461 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:46.461 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 1, 2024-04-13T03:51:46.461 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 0, 2024-04-13T03:51:46.461 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:46.461 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:46.462 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:46.462 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-04-13T03:51:46.462 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-04-13T03:51:46.462 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-04-13T03:51:46.462 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:46.462 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-04-13T03:51:46.462 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:46.462 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-04-13T03:51:46.462 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-04-13T03:51:46.463 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-04-13T03:51:46.463 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-04-13T03:51:46.463 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-04-13T03:51:46.463 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-04-13T03:51:46.463 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-04-13T03:51:46.463 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-04-13T03:51:46.463 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-04-13T03:51:46.463 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-04-13T03:51:46.463 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-04-13T03:51:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-04-13T03:51:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-04-13T03:51:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-04-13T03:51:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-04-13T03:51:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-04-13T03:51:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-04-13T03:51:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 117, 2024-04-13T03:51:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-04-13T03:51:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 27, 2024-04-13T03:51:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 10, 2024-04-13T03:51:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 10, 2024-04-13T03:51:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-04-13T03:51:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-04-13T03:51:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 0, 2024-04-13T03:51:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 2, 2024-04-13T03:51:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 0 2024-04-13T03:51:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.466 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-04-13T03:51:46.466 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 0, 2024-04-13T03:51:46.466 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:46.466 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-04-13T03:51:46.466 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-04-13T03:51:46.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.466 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-04-13T03:51:46.466 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-04-13T03:51:46.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.467 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-04-13T03:51:46.467 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:46.467 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 0, 2024-04-13T03:51:46.467 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-04-13T03:51:46.467 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 0, 2024-04-13T03:51:46.467 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 0, 2024-04-13T03:51:46.467 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 0, 2024-04-13T03:51:46.467 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:46.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.468 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:46.468 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:46.468 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-13T03:51:46.468 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:46.468 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 4467, 2024-04-13T03:51:46.468 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 127969982, 2024-04-13T03:51:46.468 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:46.468 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 4467, 2024-04-13T03:51:46.468 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:46.469 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:46.469 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 4467, 2024-04-13T03:51:46.469 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 127969982, 2024-04-13T03:51:46.469 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:46.469 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.469 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.469 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.469 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:46.469 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.470 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:46.470 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:46.470 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-13T03:51:46.470 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:46.470 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:46.470 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:46.470 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:46.470 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:46.471 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 4044, 2024-04-13T03:51:46.471 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 125301858, 2024-04-13T03:51:46.471 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 4044, 2024-04-13T03:51:46.471 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 125301858, 2024-04-13T03:51:46.471 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:46.471 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.471 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.471 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.471 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:46.472 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.472 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:46.472 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:46.472 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-04-13T03:51:46.472 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:46.472 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:46.472 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:46.472 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:46.472 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:46.473 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 4044, 2024-04-13T03:51:46.473 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 4044, 2024-04-13T03:51:46.473 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 4044, 2024-04-13T03:51:46.473 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 4044, 2024-04-13T03:51:46.473 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:46.473 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.473 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.473 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.473 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:46.474 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.474 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:46.474 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:46.474 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-13T03:51:46.474 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:46.474 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:46.474 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:46.474 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:46.474 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:46.475 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:46.475 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:46.475 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-04-13T03:51:46.475 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-04-13T03:51:46.475 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:46.475 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.475 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.475 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.475 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:46.476 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-13T03:51:46.476 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x562d4561e6e0": { 2024-04-13T03:51:46.476 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-13T03:51:46.476 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-13T03:51:46.476 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-13T03:51:46.476 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-13T03:51:46.476 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-13T03:51:46.476 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:46.476 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:46.477 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-13T03:51:46.477 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-13T03:51:46.477 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-04-13T03:51:46.477 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-04-13T03:51:46.477 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-13T03:51:46.477 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-13T03:51:46.477 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-13T03:51:46.477 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-13T03:51:46.477 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:46.478 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-13T03:51:46.478 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-13T03:51:47.156 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:46 smithi119 ceph-mon[35835]: pgmap v535: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-04-13T03:51:47.184 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-13T03:51:47.184 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-13T03:51:47.184 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-04-13T03:51:47.184 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-04-13T03:51:47.184 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-04-13T03:51:47.184 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-13T03:51:47.184 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-13T03:51:47.184 DEBUG:tasks.check_counter:Getting stats from c 2024-04-13T03:51:47.185 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-13T03:51:47.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:46 smithi016 ceph-mon[31772]: pgmap v535: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-04-13T03:51:47.642 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:46 smithi188 ceph-mon[36398]: pgmap v535: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-04-13T03:51:47.642 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:51:49.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:48 smithi016 ceph-mon[31772]: pgmap v536: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 2.7 MiB/s wr, 119 op/s 2024-04-13T03:51:49.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:48 smithi188 ceph-mon[36398]: pgmap v536: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 2.7 MiB/s wr, 119 op/s 2024-04-13T03:51:49.242 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-13T03:51:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 36564, 2024-04-13T03:51:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 33164, 2024-04-13T03:51:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 10322223, 2024-04-13T03:51:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 47532047, 2024-04-13T03:51:49.242 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 15, 2024-04-13T03:51:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2024-04-13T03:51:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 2.330158110, 2024-04-13T03:51:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.855194382, 2024-04-13T03:51:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 1.254434390, 2024-04-13T03:51:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.371219582, 2024-04-13T03:51:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 33162, 2024-04-13T03:51:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1.783988210, 2024-04-13T03:51:49.243 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000053796 2024-04-13T03:51:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 46441, 2024-04-13T03:51:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.009103907, 2024-04-13T03:51:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000196 2024-04-13T03:51:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-13T03:51:49.244 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 455184 2024-04-13T03:51:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 131962, 2024-04-13T03:51:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 168884, 2024-04-13T03:51:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 39198020, 2024-04-13T03:51:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 151292916, 2024-04-13T03:51:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 14, 2024-04-13T03:51:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 9, 2024-04-13T03:51:49.245 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 7.621381838, 2024-04-13T03:51:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 3.454582409, 2024-04-13T03:51:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 4.744596716, 2024-04-13T03:51:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.403445287, 2024-04-13T03:51:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 168880, 2024-04-13T03:51:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 7.348065638, 2024-04-13T03:51:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000043510 2024-04-13T03:51:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.246 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 164250, 2024-04-13T03:51:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.046759063, 2024-04-13T03:51:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000284 2024-04-13T03:51:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 6640, 2024-04-13T03:51:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 233776 2024-04-13T03:51:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.247 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 182752, 2024-04-13T03:51:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 136039, 2024-04-13T03:51:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 54506868, 2024-04-13T03:51:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 74840616, 2024-04-13T03:51:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 9, 2024-04-13T03:51:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 5, 2024-04-13T03:51:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 9.759286468, 2024-04-13T03:51:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 3.926694128, 2024-04-13T03:51:49.248 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 7.038860175, 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.417832601, 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 136040, 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 3.944937796, 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000028998 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 291752, 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.044323641, 2024-04-13T03:51:49.249 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000151 2024-04-13T03:51:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1988064, 2024-04-13T03:51:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 214400 2024-04-13T03:51:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-13T03:51:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-13T03:51:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-13T03:51:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.250 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-13T03:51:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-13T03:51:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-13T03:51:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 29078, 2024-04-13T03:51:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 24.354589824, 2024-04-13T03:51:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000837560 2024-04-13T03:51:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:49.251 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-13T03:51:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-13T03:51:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 16177, 2024-04-13T03:51:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1.599799644, 2024-04-13T03:51:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000098893 2024-04-13T03:51:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-13T03:51:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 64925, 2024-04-13T03:51:49.252 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 50473, 2024-04-13T03:51:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-13T03:51:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 50473, 2024-04-13T03:51:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 148.547950613, 2024-04-13T03:51:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.002943117 2024-04-13T03:51:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-13T03:51:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 14452, 2024-04-13T03:51:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 2, 2024-04-13T03:51:49.253 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 20, 2024-04-13T03:51:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 1, 2024-04-13T03:51:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 1, 2024-04-13T03:51:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 1034, 2024-04-13T03:51:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 204, 2024-04-13T03:51:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 778, 2024-04-13T03:51:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 52, 2024-04-13T03:51:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 742, 2024-04-13T03:51:49.254 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 349, 2024-04-13T03:51:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 132, 2024-04-13T03:51:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 132, 2024-04-13T03:51:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 17, 2024-04-13T03:51:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 123176, 2024-04-13T03:51:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 57107, 2024-04-13T03:51:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 14448, 2024-04-13T03:51:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 179, 2024-04-13T03:51:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:49.255 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 1986, 2024-04-13T03:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 8851, 2024-04-13T03:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2024-04-13T03:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 7, 2024-04-13T03:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 218, 2024-04-13T03:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 15, 2024-04-13T03:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 443, 2024-04-13T03:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 747, 2024-04-13T03:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 213259971, 2024-04-13T03:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 6, 2024-04-13T03:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 7, 2024-04-13T03:51:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 4837, 2024-04-13T03:51:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 34318, 2024-04-13T03:51:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:49.258 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 20765, 2024-04-13T03:51:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 9083, 2024-04-13T03:51:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 3680, 2024-04-13T03:51:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 45840 2024-04-13T03:51:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-13T03:51:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 1235, 2024-04-13T03:51:49.259 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 1407, 2024-04-13T03:51:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 764, 2024-04-13T03:51:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 1375, 2024-04-13T03:51:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 1400, 2024-04-13T03:51:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-13T03:51:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-13T03:51:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 7768, 2024-04-13T03:51:49.260 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 7768, 2024-04-13T03:51:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-13T03:51:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-13T03:51:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-13T03:51:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:49.261 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 8, 2024-04-13T03:51:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-13T03:51:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 2, 2024-04-13T03:51:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-13T03:51:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-13T03:51:49.262 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-13T03:51:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 52478, 2024-04-13T03:51:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 0, 2024-04-13T03:51:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 0, 2024-04-13T03:51:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 52478, 2024-04-13T03:51:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-13T03:51:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 0, 2024-04-13T03:51:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 73, 2024-04-13T03:51:49.263 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 0, 2024-04-13T03:51:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 0, 2024-04-13T03:51:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 73, 2024-04-13T03:51:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 6, 2024-04-13T03:51:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-13T03:51:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 0, 2024-04-13T03:51:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 4194304, 2024-04-13T03:51:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 129629000, 2024-04-13T03:51:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 0, 2024-04-13T03:51:49.264 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-13T03:51:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 14662, 2024-04-13T03:51:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 77.147577101, 2024-04-13T03:51:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.005261736 2024-04-13T03:51:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 0 2024-04-13T03:51:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-13T03:51:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 714, 2024-04-13T03:51:49.265 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 8772, 2024-04-13T03:51:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 8058, 2024-04-13T03:51:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 142, 2024-04-13T03:51:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 766, 2024-04-13T03:51:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 624, 2024-04-13T03:51:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 1045, 2024-04-13T03:51:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 13470, 2024-04-13T03:51:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 12425, 2024-04-13T03:51:49.266 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 60, 2024-04-13T03:51:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 8923, 2024-04-13T03:51:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 8863, 2024-04-13T03:51:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 72100, 2024-04-13T03:51:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 223608 2024-04-13T03:51:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-13T03:51:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 105657, 2024-04-13T03:51:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 526, 2024-04-13T03:51:49.267 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 64925, 2024-04-13T03:51:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 1388, 2024-04-13T03:51:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 12403, 2024-04-13T03:51:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-13T03:51:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 7782, 2024-04-13T03:51:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 43.596671660, 2024-04-13T03:51:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.005602245 2024-04-13T03:51:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-13T03:51:49.268 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1474, 2024-04-13T03:51:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 10.224427410, 2024-04-13T03:51:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.006936517 2024-04-13T03:51:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.269 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-13T03:51:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-13T03:51:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 13150, 2024-04-13T03:51:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 19.973460050, 2024-04-13T03:51:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001518894 2024-04-13T03:51:49.270 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-13T03:51:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.271 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.272 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.273 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.273 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:49.273 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.273 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-13T03:51:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.274 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-13T03:51:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-13T03:51:49.275 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-13T03:51:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.276 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-13T03:51:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-13T03:51:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 18046, 2024-04-13T03:51:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 26.911724451, 2024-04-13T03:51:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001491284 2024-04-13T03:51:49.277 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-13T03:51:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 2051, 2024-04-13T03:51:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 10.482835347, 2024-04-13T03:51:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.005111085 2024-04-13T03:51:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.278 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-13T03:51:49.279 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.280 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-13T03:51:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 10, 2024-04-13T03:51:49.281 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.065925506, 2024-04-13T03:51:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.006592550 2024-04-13T03:51:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:49.282 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 192, 2024-04-13T03:51:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.215965837, 2024-04-13T03:51:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001124822 2024-04-13T03:51:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-13T03:51:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.283 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-13T03:51:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.284 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-13T03:51:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-13T03:51:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 7768, 2024-04-13T03:51:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 37.076940352, 2024-04-13T03:51:49.285 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.004773035 2024-04-13T03:51:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:49.286 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-13T03:51:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2024-04-13T03:51:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 3, 2024-04-13T03:51:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 1, 2024-04-13T03:51:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 2, 2024-04-13T03:51:49.287 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-13T03:51:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 15088, 2024-04-13T03:51:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 7544, 2024-04-13T03:51:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 968, 2024-04-13T03:51:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-13T03:51:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:49.288 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:49.289 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:49.290 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:49.291 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-13T03:51:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 497316, 2024-04-13T03:51:49.292 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 664, 2024-04-13T03:51:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 352, 2024-04-13T03:51:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 4, 2024-04-13T03:51:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-13T03:51:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-13T03:51:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:49.293 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 186, 2024-04-13T03:51:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-13T03:51:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 2884863, 2024-04-13T03:51:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 26674, 2024-04-13T03:51:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-13T03:51:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-13T03:51:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-13T03:51:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-13T03:51:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-13T03:51:49.295 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 32706, 2024-04-13T03:51:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 127208291, 2024-04-13T03:51:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 28, 2024-04-13T03:51:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 32706, 2024-04-13T03:51:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-13T03:51:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 32706, 2024-04-13T03:51:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 265.104134573, 2024-04-13T03:51:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.008105672 2024-04-13T03:51:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.296 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-13T03:51:49.297 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-13T03:51:49.297 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 32706, 2024-04-13T03:51:49.297 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 33781 2024-04-13T03:51:49.297 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.297 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 32706, 2024-04-13T03:51:49.297 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 673, 2024-04-13T03:51:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 32033, 2024-04-13T03:51:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-13T03:51:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-13T03:51:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 0, 2024-04-13T03:51:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 10, 2024-04-13T03:51:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 670, 2024-04-13T03:51:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 15332, 2024-04-13T03:51:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 403, 2024-04-13T03:51:49.298 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-13T03:51:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-13T03:51:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-13T03:51:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 1, 2024-04-13T03:51:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 15585, 2024-04-13T03:51:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-13T03:51:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 0, 2024-04-13T03:51:49.299 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 20, 2024-04-13T03:51:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-13T03:51:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-13T03:51:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-13T03:51:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-13T03:51:49.300 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-13T03:51:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-13T03:51:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-13T03:51:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-13T03:51:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-13T03:51:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-13T03:51:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-13T03:51:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-13T03:51:49.301 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-13T03:51:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-13T03:51:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-13T03:51:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-13T03:51:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-13T03:51:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-13T03:51:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-13T03:51:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-13T03:51:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-13T03:51:49.302 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 117, 2024-04-13T03:51:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-13T03:51:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 27, 2024-04-13T03:51:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 12, 2024-04-13T03:51:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 12, 2024-04-13T03:51:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2024-04-13T03:51:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-13T03:51:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 1329, 2024-04-13T03:51:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 5, 2024-04-13T03:51:49.303 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 426 2024-04-13T03:51:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-13T03:51:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 1, 2024-04-13T03:51:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-13T03:51:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-13T03:51:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-13T03:51:49.304 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 0 2024-04-13T03:51:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-13T03:51:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 10, 2024-04-13T03:51:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-13T03:51:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 5, 2024-04-13T03:51:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 15536, 2024-04-13T03:51:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 7768, 2024-04-13T03:51:49.305 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-13T03:51:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 351274, 2024-04-13T03:51:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 73900780, 2024-04-13T03:51:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:49.306 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 351274, 2024-04-13T03:51:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 351274, 2024-04-13T03:51:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 73900780, 2024-04-13T03:51:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.307 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-13T03:51:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:49.308 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 32706, 2024-04-13T03:51:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 128056502, 2024-04-13T03:51:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 32706, 2024-04-13T03:51:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 128056502, 2024-04-13T03:51:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.309 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-13T03:51:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:49.310 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 32706, 2024-04-13T03:51:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 32706, 2024-04-13T03:51:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 32706, 2024-04-13T03:51:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 32706, 2024-04-13T03:51:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.311 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-13T03:51:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 7768, 2024-04-13T03:51:49.312 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 848272, 2024-04-13T03:51:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 7768, 2024-04-13T03:51:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 670, 2024-04-13T03:51:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 848272, 2024-04-13T03:51:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.313 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x55e41b6ec500": { 2024-04-13T03:51:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 109238, 2024-04-13T03:51:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-13T03:51:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 52478, 2024-04-13T03:51:49.314 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 125434696, 2024-04-13T03:51:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 52478, 2024-04-13T03:51:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 14662, 2024-04-13T03:51:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 125325458, 2024-04-13T03:51:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:49.315 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:49.316 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:49.316 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:49.316 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:49.316 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-13T03:51:49.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:48 smithi119 ceph-mon[35835]: pgmap v536: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 2.7 MiB/s wr, 119 op/s 2024-04-13T03:51:49.946 INFO:tasks.check_counter:Daemon mds.c mds.exported=7 2024-04-13T03:51:49.946 INFO:tasks.check_counter:Daemon mds.c mds.imported=15 2024-04-13T03:51:49.946 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=1235 2024-04-13T03:51:49.946 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=1407 2024-04-13T03:51:49.946 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-13T03:51:49.946 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-13T03:51:49.946 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-13T03:51:49.947 DEBUG:tasks.check_counter:Getting stats from f 2024-04-13T03:51:49.947 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-13T03:51:50.193 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:51:50.314 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:49 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:50.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:49 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:50.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:49 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:51.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:50 smithi119 ceph-mon[35835]: pgmap v537: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.9 MiB/s wr, 75 op/s 2024-04-13T03:51:51.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:50 smithi119 ceph-mon[35835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:50 smithi016 ceph-mon[31772]: pgmap v537: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.9 MiB/s wr, 75 op/s 2024-04-13T03:51:51.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:50 smithi016 ceph-mon[31772]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:51.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:50 smithi188 ceph-mon[36398]: pgmap v537: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 1.9 MiB/s wr, 75 op/s 2024-04-13T03:51:51.373 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:50 smithi188 ceph-mon[36398]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-13T03:51:52.062 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-13T03:51:52.062 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:52.062 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 2842, 2024-04-13T03:51:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 2781, 2024-04-13T03:51:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 169517073, 2024-04-13T03:51:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 978564, 2024-04-13T03:51:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 7, 2024-04-13T03:51:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 3, 2024-04-13T03:51:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.357683563, 2024-04-13T03:51:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.081473197, 2024-04-13T03:51:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.192391309, 2024-04-13T03:51:52.063 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.074430328, 2024-04-13T03:51:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 2781, 2024-04-13T03:51:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.093871475, 2024-04-13T03:51:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000033754 2024-04-13T03:51:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.064 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1984000, 2024-04-13T03:51:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 210272 2024-04-13T03:51:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 926, 2024-04-13T03:51:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 1176, 2024-04-13T03:51:52.065 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 142843272, 2024-04-13T03:51:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 892778, 2024-04-13T03:51:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 9, 2024-04-13T03:51:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 7, 2024-04-13T03:51:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.147426375, 2024-04-13T03:51:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.028735971, 2024-04-13T03:51:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.089298419, 2024-04-13T03:51:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.021308038, 2024-04-13T03:51:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:52.066 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1173, 2024-04-13T03:51:52.067 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.050593229, 2024-04-13T03:51:52.067 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000043131 2024-04-13T03:51:52.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.067 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:52.067 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.067 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.067 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.067 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.067 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-13T03:51:52.068 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 598880 2024-04-13T03:51:52.068 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.068 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:52.068 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 1539, 2024-04-13T03:51:52.068 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 1541, 2024-04-13T03:51:52.068 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 235286468, 2024-04-13T03:51:52.068 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 489409, 2024-04-13T03:51:52.068 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 6, 2024-04-13T03:51:52.069 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 5, 2024-04-13T03:51:52.069 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.222297329, 2024-04-13T03:51:52.069 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.037187239, 2024-04-13T03:51:52.069 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.136946933, 2024-04-13T03:51:52.069 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.037833951, 2024-04-13T03:51:52.069 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:52.069 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1539, 2024-04-13T03:51:52.069 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.057089441, 2024-04-13T03:51:52.069 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000037095 2024-04-13T03:51:52.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.070 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:52.070 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.070 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.070 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.070 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.070 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-13T03:51:52.070 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 0 2024-04-13T03:51:52.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.071 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-13T03:51:52.071 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-13T03:51:52.071 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-13T03:51:52.071 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.071 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-13T03:51:52.071 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-13T03:51:52.071 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-13T03:51:52.071 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 7987, 2024-04-13T03:51:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.184317641, 2024-04-13T03:51:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000023077 2024-04-13T03:51:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-13T03:51:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-13T03:51:52.072 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-13T03:51:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 0, 2024-04-13T03:51:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 0, 2024-04-13T03:51:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-13T03:51:52.073 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-13T03:51:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 0, 2024-04-13T03:51:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 0, 2024-04-13T03:51:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:52.074 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 0, 2024-04-13T03:51:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 0, 2024-04-13T03:51:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 0, 2024-04-13T03:51:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 12846, 2024-04-13T03:51:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 1288, 2024-04-13T03:51:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 11558, 2024-04-13T03:51:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 0, 2024-04-13T03:51:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 11007, 2024-04-13T03:51:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 0, 2024-04-13T03:51:52.075 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 0, 2024-04-13T03:51:52.076 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 0, 2024-04-13T03:51:52.076 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 13, 2024-04-13T03:51:52.076 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 0, 2024-04-13T03:51:52.076 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 0, 2024-04-13T03:51:52.076 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 0, 2024-04-13T03:51:52.076 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 0, 2024-04-13T03:51:52.076 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:52.076 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:52.076 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 0, 2024-04-13T03:51:52.077 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 0, 2024-04-13T03:51:52.077 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2024-04-13T03:51:52.077 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 0, 2024-04-13T03:51:52.077 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 0, 2024-04-13T03:51:52.077 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 0, 2024-04-13T03:51:52.077 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 0, 2024-04-13T03:51:52.077 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:52.077 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:52.077 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:52.078 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 747, 2024-04-13T03:51:52.078 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 222986783, 2024-04-13T03:51:52.078 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 6, 2024-04-13T03:51:52.078 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:52.078 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:52.078 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:52.078 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:52.078 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:52.078 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:52.079 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:52.079 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:52.079 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 0, 2024-04-13T03:51:52.079 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 0, 2024-04-13T03:51:52.079 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 0, 2024-04-13T03:51:52.079 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:52.079 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:52.080 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:52.080 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 0, 2024-04-13T03:51:52.080 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 0, 2024-04-13T03:51:52.080 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 0, 2024-04-13T03:51:52.080 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 0 2024-04-13T03:51:52.080 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.080 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-13T03:51:52.080 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 0, 2024-04-13T03:51:52.081 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 0, 2024-04-13T03:51:52.081 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 0, 2024-04-13T03:51:52.081 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 0, 2024-04-13T03:51:52.081 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 0, 2024-04-13T03:51:52.081 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-13T03:51:52.081 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-13T03:51:52.081 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:52.081 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 0, 2024-04-13T03:51:52.081 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 0, 2024-04-13T03:51:52.082 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:52.082 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-13T03:51:52.082 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:52.082 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:52.082 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:52.082 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-13T03:51:52.082 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-13T03:51:52.083 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:52.083 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:52.083 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:52.083 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 0, 2024-04-13T03:51:52.083 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-13T03:51:52.083 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 0, 2024-04-13T03:51:52.083 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:52.083 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-13T03:51:52.083 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.084 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-13T03:51:52.084 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-13T03:51:52.084 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 0, 2024-04-13T03:51:52.084 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 0, 2024-04-13T03:51:52.084 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 0, 2024-04-13T03:51:52.084 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 76478, 2024-04-13T03:51:52.084 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-13T03:51:52.084 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 0, 2024-04-13T03:51:52.084 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 0, 2024-04-13T03:51:52.085 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 0, 2024-04-13T03:51:52.085 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 0, 2024-04-13T03:51:52.085 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 143, 2024-04-13T03:51:52.085 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 8, 2024-04-13T03:51:52.085 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-13T03:51:52.085 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 0, 2024-04-13T03:51:52.085 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 42951514, 2024-04-13T03:51:52.085 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 548501595, 2024-04-13T03:51:52.085 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 548457126, 2024-04-13T03:51:52.086 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-13T03:51:52.086 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.086 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.086 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.086 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 76478 2024-04-13T03:51:52.086 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.086 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-13T03:51:52.086 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 1099, 2024-04-13T03:51:52.087 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 13374, 2024-04-13T03:51:52.087 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 12275, 2024-04-13T03:51:52.087 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 68, 2024-04-13T03:51:52.087 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 157, 2024-04-13T03:51:52.087 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 89, 2024-04-13T03:51:52.087 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 12846, 2024-04-13T03:51:52.087 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 14144, 2024-04-13T03:51:52.087 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 1298, 2024-04-13T03:51:52.087 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 0, 2024-04-13T03:51:52.088 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 0, 2024-04-13T03:51:52.088 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 0, 2024-04-13T03:51:52.088 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 69536, 2024-04-13T03:51:52.088 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 215416 2024-04-13T03:51:52.088 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.088 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-13T03:51:52.088 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 0, 2024-04-13T03:51:52.088 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 0, 2024-04-13T03:51:52.088 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 0, 2024-04-13T03:51:52.089 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 0, 2024-04-13T03:51:52.089 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 0, 2024-04-13T03:51:52.089 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-13T03:51:52.089 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.089 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.089 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.089 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.090 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-13T03:51:52.090 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.090 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.090 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.090 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.090 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:52.090 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.090 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.091 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.091 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-13T03:51:52.091 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.091 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.091 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.091 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.091 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-13T03:51:52.091 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-13T03:51:52.092 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.093 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.093 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.093 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-13T03:51:52.093 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.093 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.093 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.093 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.094 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:52.094 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.094 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.094 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.094 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.094 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:52.094 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.094 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.095 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.095 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.095 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-13T03:51:52.095 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.095 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.095 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.096 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-13T03:51:52.096 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.096 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.096 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.096 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.096 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-13T03:51:52.097 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.097 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.097 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.097 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-13T03:51:52.097 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.097 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.097 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.097 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.098 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-13T03:51:52.098 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.098 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.098 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.098 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.098 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-13T03:51:52.098 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.099 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.099 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.099 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-13T03:51:52.099 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.099 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.099 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.099 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.099 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:52.100 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.100 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.100 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.100 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:52.100 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.100 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.100 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.100 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.101 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.102 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:52.102 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.102 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.102 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.102 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.102 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-13T03:51:52.102 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.102 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.103 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.103 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:52.103 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.103 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.103 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.103 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.103 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:52.103 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.104 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.104 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.104 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-13T03:51:52.104 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.104 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.104 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.104 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.105 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-13T03:51:52.105 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.105 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.105 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.105 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.105 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-13T03:51:52.105 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.105 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.105 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.106 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-13T03:51:52.106 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.106 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.106 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.106 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.106 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:52.106 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:52.106 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:52.107 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.107 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.107 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.107 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:52.107 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.107 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-13T03:51:52.107 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2024-04-13T03:51:52.107 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 6, 2024-04-13T03:51:52.107 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 4, 2024-04-13T03:51:52.108 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 4, 2024-04-13T03:51:52.108 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-13T03:51:52.108 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 0, 2024-04-13T03:51:52.108 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 0, 2024-04-13T03:51:52.108 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 4263, 2024-04-13T03:51:52.108 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:52.108 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.108 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-13T03:51:52.108 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:52.109 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:52.109 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:52.109 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:52.109 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:52.109 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:52.109 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:52.109 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:52.109 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:52.109 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:52.110 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:52.110 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:52.110 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:52.110 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:52.110 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:52.110 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:52.110 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:52.110 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:52.110 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:52.111 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:52.111 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:52.111 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:52.111 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:52.111 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:52.111 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:52.111 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:52.111 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:52.111 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 288716, 2024-04-13T03:51:52.112 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 1373, 2024-04-13T03:51:52.113 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2024-04-13T03:51:52.113 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2024-04-13T03:51:52.113 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-13T03:51:52.113 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-13T03:51:52.113 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:52.113 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:52.113 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:52.113 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 186, 2024-04-13T03:51:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-13T03:51:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 12062477, 2024-04-13T03:51:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 81691, 2024-04-13T03:51:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:52.115 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-13T03:51:52.115 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:52.115 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-13T03:51:52.115 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.115 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-13T03:51:52.115 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-13T03:51:52.115 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-13T03:51:52.115 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 4879, 2024-04-13T03:51:52.115 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 26, 2024-04-13T03:51:52.116 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 4, 2024-04-13T03:51:52.116 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 4879, 2024-04-13T03:51:52.116 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-13T03:51:52.116 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 4879, 2024-04-13T03:51:52.116 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 11.225793584, 2024-04-13T03:51:52.116 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.002300839 2024-04-13T03:51:52.116 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.116 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-13T03:51:52.116 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-13T03:51:52.117 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 4879, 2024-04-13T03:51:52.117 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 4880 2024-04-13T03:51:52.117 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.117 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 4879, 2024-04-13T03:51:52.117 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 4879, 2024-04-13T03:51:52.117 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 0, 2024-04-13T03:51:52.117 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-13T03:51:52.117 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-13T03:51:52.117 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 2733, 2024-04-13T03:51:52.118 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 0, 2024-04-13T03:51:52.118 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 2143, 2024-04-13T03:51:52.118 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 0, 2024-04-13T03:51:52.118 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 0, 2024-04-13T03:51:52.118 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-13T03:51:52.118 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-13T03:51:52.118 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-13T03:51:52.118 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 0, 2024-04-13T03:51:52.119 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 0, 2024-04-13T03:51:52.119 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-13T03:51:52.119 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:52.119 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:52.119 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 2, 2024-04-13T03:51:52.119 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 0, 2024-04-13T03:51:52.119 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:52.119 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:52.119 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:52.120 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-13T03:51:52.120 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-13T03:51:52.120 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-13T03:51:52.120 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:52.120 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-13T03:51:52.120 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:52.120 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-13T03:51:52.120 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-13T03:51:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-13T03:51:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-13T03:51:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-13T03:51:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-13T03:51:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-13T03:51:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-13T03:51:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-13T03:51:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-13T03:51:52.121 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-13T03:51:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-13T03:51:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-13T03:51:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-13T03:51:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-13T03:51:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-13T03:51:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-13T03:51:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 117, 2024-04-13T03:51:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-13T03:51:52.122 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 27, 2024-04-13T03:51:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 11, 2024-04-13T03:51:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 11, 2024-04-13T03:51:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2024-04-13T03:51:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-13T03:51:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 0, 2024-04-13T03:51:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 2, 2024-04-13T03:51:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 0 2024-04-13T03:51:52.123 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-13T03:51:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 0, 2024-04-13T03:51:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-13T03:51:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-13T03:51:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-13T03:51:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 0 2024-04-13T03:51:52.124 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-13T03:51:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 0, 2024-04-13T03:51:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-13T03:51:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 0, 2024-04-13T03:51:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 0, 2024-04-13T03:51:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 0, 2024-04-13T03:51:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:52.125 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.126 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:52.126 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:52.126 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-13T03:51:52.126 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:52.126 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 5303, 2024-04-13T03:51:52.126 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 547145120, 2024-04-13T03:51:52.126 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:52.126 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 5303, 2024-04-13T03:51:52.126 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:52.127 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:52.127 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 5303, 2024-04-13T03:51:52.127 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 547145120, 2024-04-13T03:51:52.127 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:52.127 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.127 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.127 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.127 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:52.127 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.128 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:52.128 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:52.128 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-13T03:51:52.128 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:52.128 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:52.128 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:52.128 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:52.128 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:52.128 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 4879, 2024-04-13T03:51:52.129 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 544321373, 2024-04-13T03:51:52.129 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 4879, 2024-04-13T03:51:52.129 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 544321373, 2024-04-13T03:51:52.129 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:52.129 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.129 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.129 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.129 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:52.129 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.130 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:52.130 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:52.130 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-13T03:51:52.130 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:52.130 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:52.130 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:52.130 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:52.130 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:52.130 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 4879, 2024-04-13T03:51:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 4879, 2024-04-13T03:51:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 4879, 2024-04-13T03:51:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 4879, 2024-04-13T03:51:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:52.131 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-13T03:51:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:52.132 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:52.133 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:52.133 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2024-04-13T03:51:52.133 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2024-04-13T03:51:52.133 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:52.133 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.133 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.133 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.133 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:52.134 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:52.134 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x556ff66126e0": { 2024-04-13T03:51:52.134 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:52.134 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-13T03:51:52.134 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:52.134 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:52.134 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:52.134 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:52.134 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:52.135 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:52.135 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:52.135 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2024-04-13T03:51:52.135 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2024-04-13T03:51:52.135 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:52.135 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:52.135 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:52.135 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:52.135 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:52.136 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:52.136 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-13T03:51:52.824 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-13T03:51:52.825 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-13T03:51:52.825 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-13T03:51:52.825 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-13T03:51:52.825 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-13T03:51:52.825 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-13T03:51:52.825 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-13T03:51:52.825 DEBUG:tasks.check_counter:Getting stats from i 2024-04-13T03:51:52.826 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-13T03:51:53.066 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:51:53.321 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:53 smithi119 ceph-mon[35835]: pgmap v538: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 2.0 MiB/s wr, 82 op/s 2024-04-13T03:51:53.331 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:53 smithi188 ceph-mon[36398]: pgmap v538: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 2.0 MiB/s wr, 82 op/s 2024-04-13T03:51:53.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:53 smithi016 ceph-mon[31772]: pgmap v538: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 2.0 MiB/s wr, 82 op/s 2024-04-13T03:51:54.811 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-13T03:51:54.812 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:54.812 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 118144, 2024-04-13T03:51:54.812 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 172325, 2024-04-13T03:51:54.812 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 36475457, 2024-04-13T03:51:54.812 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 174296260, 2024-04-13T03:51:54.812 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 18, 2024-04-13T03:51:54.812 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 4, 2024-04-13T03:51:54.813 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 7.182575966, 2024-04-13T03:51:54.813 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 3.299280832, 2024-04-13T03:51:54.813 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 4.179580967, 2024-04-13T03:51:54.813 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.529564172, 2024-04-13T03:51:54.813 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:54.813 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 172324, 2024-04-13T03:51:54.813 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 7.485179647, 2024-04-13T03:51:54.813 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000043436 2024-04-13T03:51:54.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.814 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:54.814 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 136078, 2024-04-13T03:51:54.814 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.042685400, 2024-04-13T03:51:54.814 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000313 2024-04-13T03:51:54.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.814 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 5120, 2024-04-13T03:51:54.815 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 459008 2024-04-13T03:51:54.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.815 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:54.815 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 18653, 2024-04-13T03:51:54.815 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 18730, 2024-04-13T03:51:54.815 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 4712278, 2024-04-13T03:51:54.815 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 97264069, 2024-04-13T03:51:54.816 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 14, 2024-04-13T03:51:54.816 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 8, 2024-04-13T03:51:54.816 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 1.962885297, 2024-04-13T03:51:54.816 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.756129607, 2024-04-13T03:51:54.816 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.337060728, 2024-04-13T03:51:54.817 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.765226277, 2024-04-13T03:51:54.817 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:54.817 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 18727, 2024-04-13T03:51:54.817 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 1.059609796, 2024-04-13T03:51:54.817 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000056581 2024-04-13T03:51:54.817 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.817 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:54.818 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 80, 2024-04-13T03:51:54.818 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000015493, 2024-04-13T03:51:54.818 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000193 2024-04-13T03:51:54.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.818 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 5632, 2024-04-13T03:51:54.818 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 157936 2024-04-13T03:51:54.818 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.819 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:54.819 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 239742, 2024-04-13T03:51:54.819 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 215727, 2024-04-13T03:51:54.819 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 67493156, 2024-04-13T03:51:54.819 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 67615283, 2024-04-13T03:51:54.819 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 8, 2024-04-13T03:51:54.819 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 7, 2024-04-13T03:51:54.820 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 12.599185097, 2024-04-13T03:51:54.820 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 5.141984554, 2024-04-13T03:51:54.820 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 9.396157773, 2024-04-13T03:51:54.820 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.248676135, 2024-04-13T03:51:54.820 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:54.820 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 215726, 2024-04-13T03:51:54.820 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 8.456464494, 2024-04-13T03:51:54.821 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000039200 2024-04-13T03:51:54.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.821 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:54.821 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 436680, 2024-04-13T03:51:54.821 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.066159106, 2024-04-13T03:51:54.821 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000151 2024-04-13T03:51:54.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.821 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 2000016, 2024-04-13T03:51:54.822 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 214400 2024-04-13T03:51:54.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.822 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-13T03:51:54.822 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-13T03:51:54.822 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-13T03:51:54.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.822 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-13T03:51:54.823 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-13T03:51:54.823 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-13T03:51:54.823 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 27388, 2024-04-13T03:51:54.823 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 33.931202804, 2024-04-13T03:51:54.823 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001238907 2024-04-13T03:51:54.823 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:54.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.823 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:54.824 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-13T03:51:54.824 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-13T03:51:54.824 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 19526, 2024-04-13T03:51:54.824 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 2.436910785, 2024-04-13T03:51:54.824 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000124803 2024-04-13T03:51:54.824 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:54.824 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.824 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-13T03:51:54.825 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 73199, 2024-04-13T03:51:54.825 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 59653, 2024-04-13T03:51:54.825 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-13T03:51:54.825 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 59653, 2024-04-13T03:51:54.825 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 298.338486852, 2024-04-13T03:51:54.825 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.005001231 2024-04-13T03:51:54.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.826 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-13T03:51:54.826 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 13546, 2024-04-13T03:51:54.826 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 0, 2024-04-13T03:51:54.826 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:54.826 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 38, 2024-04-13T03:51:54.826 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 3, 2024-04-13T03:51:54.827 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 3, 2024-04-13T03:51:54.827 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 1095, 2024-04-13T03:51:54.827 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 324, 2024-04-13T03:51:54.827 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 609, 2024-04-13T03:51:54.827 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 162, 2024-04-13T03:51:54.828 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 631, 2024-04-13T03:51:54.828 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 627, 2024-04-13T03:51:54.828 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 83, 2024-04-13T03:51:54.828 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 83, 2024-04-13T03:51:54.828 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 15, 2024-04-13T03:51:54.828 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 129038, 2024-04-13T03:51:54.829 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 63019, 2024-04-13T03:51:54.829 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 13541, 2024-04-13T03:51:54.829 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 305, 2024-04-13T03:51:54.829 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:54.829 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:54.829 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 1702, 2024-04-13T03:51:54.829 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 6796, 2024-04-13T03:51:54.829 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2024-04-13T03:51:54.830 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 7, 2024-04-13T03:51:54.830 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 357, 2024-04-13T03:51:54.830 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 7, 2024-04-13T03:51:54.830 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 533, 2024-04-13T03:51:54.830 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:54.830 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:54.830 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:54.830 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 747, 2024-04-13T03:51:54.831 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 213259971, 2024-04-13T03:51:54.831 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 6, 2024-04-13T03:51:54.831 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:54.831 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:54.831 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:54.831 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:54.831 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:54.832 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:54.832 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:54.832 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:54.832 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 11, 2024-04-13T03:51:54.832 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 12090, 2024-04-13T03:51:54.832 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 43265, 2024-04-13T03:51:54.832 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:54.832 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:54.833 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:54.833 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 25370, 2024-04-13T03:51:54.833 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 6929, 2024-04-13T03:51:54.833 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 3698, 2024-04-13T03:51:54.833 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 45628 2024-04-13T03:51:54.833 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.833 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-13T03:51:54.834 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 735, 2024-04-13T03:51:54.834 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 1695, 2024-04-13T03:51:54.834 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 936, 2024-04-13T03:51:54.834 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 1677, 2024-04-13T03:51:54.834 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 950, 2024-04-13T03:51:54.834 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-13T03:51:54.834 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-13T03:51:54.834 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:54.835 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 9652, 2024-04-13T03:51:54.835 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 9652, 2024-04-13T03:51:54.835 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:54.835 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-13T03:51:54.835 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:54.835 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:54.835 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:54.835 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-13T03:51:54.836 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-13T03:51:54.836 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:54.836 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:54.836 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:54.836 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 7, 2024-04-13T03:51:54.836 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-13T03:51:54.836 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 6, 2024-04-13T03:51:54.836 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:54.837 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-13T03:51:54.837 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.837 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-13T03:51:54.837 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-13T03:51:54.837 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 51534, 2024-04-13T03:51:54.837 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 0, 2024-04-13T03:51:54.837 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 0, 2024-04-13T03:51:54.837 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 51534, 2024-04-13T03:51:54.837 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-13T03:51:54.838 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 0, 2024-04-13T03:51:54.838 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 74, 2024-04-13T03:51:54.838 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 0, 2024-04-13T03:51:54.838 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 0, 2024-04-13T03:51:54.838 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 74, 2024-04-13T03:51:54.838 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 6, 2024-04-13T03:51:54.838 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-13T03:51:54.839 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 0, 2024-04-13T03:51:54.839 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 4194304, 2024-04-13T03:51:54.839 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 156978556, 2024-04-13T03:51:54.839 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 0, 2024-04-13T03:51:54.839 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-13T03:51:54.839 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 14097, 2024-04-13T03:51:54.839 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 82.513519781, 2024-04-13T03:51:54.840 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.005853268 2024-04-13T03:51:54.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.840 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 0 2024-04-13T03:51:54.840 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.840 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-13T03:51:54.840 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 552, 2024-04-13T03:51:54.840 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 10612, 2024-04-13T03:51:54.840 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 10060, 2024-04-13T03:51:54.841 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 130, 2024-04-13T03:51:54.841 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 889, 2024-04-13T03:51:54.841 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 759, 2024-04-13T03:51:54.841 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 1095, 2024-04-13T03:51:54.841 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 17787, 2024-04-13T03:51:54.841 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 16692, 2024-04-13T03:51:54.841 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 83, 2024-04-13T03:51:54.841 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 10876, 2024-04-13T03:51:54.842 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 10793, 2024-04-13T03:51:54.842 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 73644, 2024-04-13T03:51:54.842 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 223608 2024-04-13T03:51:54.842 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.842 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-13T03:51:54.842 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 112534, 2024-04-13T03:51:54.842 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 1026, 2024-04-13T03:51:54.843 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 73199, 2024-04-13T03:51:54.843 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 1403, 2024-04-13T03:51:54.843 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 10040, 2024-04-13T03:51:54.843 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-13T03:51:54.843 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 9623, 2024-04-13T03:51:54.843 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 43.450042450, 2024-04-13T03:51:54.843 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.004515228 2024-04-13T03:51:54.843 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.844 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-13T03:51:54.844 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1447, 2024-04-13T03:51:54.844 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 21.795321081, 2024-04-13T03:51:54.844 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.015062419 2024-04-13T03:51:54.844 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.844 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:54.844 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.845 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.845 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.845 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-13T03:51:54.845 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.845 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.845 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.845 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.846 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-13T03:51:54.846 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 18953, 2024-04-13T03:51:54.846 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 47.176737689, 2024-04-13T03:51:54.846 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.002489143 2024-04-13T03:51:54.846 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.846 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:54.846 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.847 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.847 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.847 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-13T03:51:54.847 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 2, 2024-04-13T03:51:54.847 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000236761, 2024-04-13T03:51:54.847 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000118380 2024-04-13T03:51:54.847 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.848 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-13T03:51:54.848 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.848 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.848 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.848 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.848 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:54.848 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.848 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.849 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.849 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.849 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:54.849 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.849 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.850 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.850 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.850 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-13T03:51:54.850 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.850 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.851 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.851 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-13T03:51:54.851 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.851 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.851 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.851 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.851 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-13T03:51:54.852 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.852 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.852 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.852 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.852 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-13T03:51:54.852 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.852 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.852 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.853 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-13T03:51:54.853 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.853 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.853 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.853 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.853 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-13T03:51:54.854 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 16878, 2024-04-13T03:51:54.854 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 73.967250408, 2024-04-13T03:51:54.854 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.004382465 2024-04-13T03:51:54.854 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.854 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-13T03:51:54.854 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 2449, 2024-04-13T03:51:54.854 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 76.545197851, 2024-04-13T03:51:54.855 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.031255695 2024-04-13T03:51:54.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.855 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:54.855 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.855 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.855 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.855 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.855 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:54.856 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.856 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.856 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.856 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.856 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-13T03:51:54.856 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.856 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.856 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.857 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:54.857 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.857 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.857 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.857 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.857 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:54.857 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.858 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.858 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.858 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.858 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-13T03:51:54.858 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 9, 2024-04-13T03:51:54.858 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.238965812, 2024-04-13T03:51:54.858 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.026551756 2024-04-13T03:51:54.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.859 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:54.859 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.859 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.859 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.859 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.859 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:54.859 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 640, 2024-04-13T03:51:54.860 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.472636416, 2024-04-13T03:51:54.860 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000738494 2024-04-13T03:51:54.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.860 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-13T03:51:54.860 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.860 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.860 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.860 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.861 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-13T03:51:54.861 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.861 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.861 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.861 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.861 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-13T03:51:54.861 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.861 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.862 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.862 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-13T03:51:54.862 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 9652, 2024-04-13T03:51:54.862 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 34.692098384, 2024-04-13T03:51:54.862 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.003594291 2024-04-13T03:51:54.862 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.863 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:54.863 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:54.863 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:54.863 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.863 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.863 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.863 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:54.863 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.864 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-13T03:51:54.864 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 2, 2024-04-13T03:51:54.864 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 5, 2024-04-13T03:51:54.864 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 3, 2024-04-13T03:51:54.864 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 2, 2024-04-13T03:51:54.864 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-13T03:51:54.864 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 18189, 2024-04-13T03:51:54.865 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 9094, 2024-04-13T03:51:54.865 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 1022, 2024-04-13T03:51:54.865 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:54.865 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.865 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-13T03:51:54.865 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:54.865 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:54.865 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:54.865 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:54.866 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:54.866 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:54.866 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:54.866 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:54.866 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:54.866 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:54.866 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:54.867 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:54.867 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:54.867 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:54.867 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:54.867 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:54.867 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:54.867 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:54.867 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:54.868 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:54.868 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:54.868 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:54.868 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:54.868 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:54.868 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:54.868 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:54.868 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:54.869 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:54.869 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:54.869 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:54.869 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:54.869 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-13T03:51:54.869 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:54.869 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:54.869 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:54.870 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:54.870 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 345939, 2024-04-13T03:51:54.870 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 573, 2024-04-13T03:51:54.870 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 352, 2024-04-13T03:51:54.870 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 4, 2024-04-13T03:51:54.870 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-13T03:51:54.870 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-13T03:51:54.871 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:54.871 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:54.871 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:54.871 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:54.871 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:54.871 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 186, 2024-04-13T03:51:54.871 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:54.871 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:54.872 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:54.872 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-13T03:51:54.872 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 2631446, 2024-04-13T03:51:54.872 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 24469, 2024-04-13T03:51:54.872 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:54.872 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-13T03:51:54.872 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:54.873 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-13T03:51:54.873 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.873 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-13T03:51:54.873 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-13T03:51:54.873 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-13T03:51:54.873 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 35969, 2024-04-13T03:51:54.873 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 155230384, 2024-04-13T03:51:54.873 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 28, 2024-04-13T03:51:54.874 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 35969, 2024-04-13T03:51:54.874 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-13T03:51:54.874 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 35969, 2024-04-13T03:51:54.874 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 304.117416224, 2024-04-13T03:51:54.874 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.008454986 2024-04-13T03:51:54.874 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.874 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-13T03:51:54.875 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-13T03:51:54.875 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 35969, 2024-04-13T03:51:54.875 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 37020 2024-04-13T03:51:54.875 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.875 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 35969, 2024-04-13T03:51:54.875 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 681, 2024-04-13T03:51:54.875 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 35288, 2024-04-13T03:51:54.876 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-13T03:51:54.876 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-13T03:51:54.876 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 0, 2024-04-13T03:51:54.876 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 10, 2024-04-13T03:51:54.876 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 681, 2024-04-13T03:51:54.876 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 14778, 2024-04-13T03:51:54.876 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 409, 2024-04-13T03:51:54.877 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-13T03:51:54.877 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-13T03:51:54.877 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-13T03:51:54.877 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 3, 2024-04-13T03:51:54.877 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 19376, 2024-04-13T03:51:54.877 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-13T03:51:54.877 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:54.877 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:54.878 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 0, 2024-04-13T03:51:54.878 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 20, 2024-04-13T03:51:54.878 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:54.878 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:54.878 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:54.878 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-13T03:51:54.878 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-13T03:51:54.879 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-13T03:51:54.879 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:54.879 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-13T03:51:54.879 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:54.879 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-13T03:51:54.879 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-13T03:51:54.879 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-13T03:51:54.880 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-13T03:51:54.880 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-13T03:51:54.880 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-13T03:51:54.880 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-13T03:51:54.880 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-13T03:51:54.880 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-13T03:51:54.880 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-13T03:51:54.880 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-13T03:51:54.881 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-13T03:51:54.881 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-13T03:51:54.881 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-13T03:51:54.881 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-13T03:51:54.881 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-13T03:51:54.881 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-13T03:51:54.881 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 117, 2024-04-13T03:51:54.881 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-13T03:51:54.882 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 27, 2024-04-13T03:51:54.882 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 12, 2024-04-13T03:51:54.882 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 12, 2024-04-13T03:51:54.882 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 0, 2024-04-13T03:51:54.882 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-13T03:51:54.882 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 1329, 2024-04-13T03:51:54.882 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 0, 2024-04-13T03:51:54.883 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 414 2024-04-13T03:51:54.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.883 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-13T03:51:54.883 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 1, 2024-04-13T03:51:54.883 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:54.883 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-13T03:51:54.883 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-13T03:51:54.883 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.884 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-13T03:51:54.884 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 0 2024-04-13T03:51:54.884 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.884 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-13T03:51:54.884 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:54.884 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 10, 2024-04-13T03:51:54.884 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-13T03:51:54.885 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 5, 2024-04-13T03:51:54.885 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 19304, 2024-04-13T03:51:54.885 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 9652, 2024-04-13T03:51:54.885 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:54.885 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.885 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:54.885 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:54.885 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-13T03:51:54.886 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:54.886 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 376535, 2024-04-13T03:51:54.886 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 76457877, 2024-04-13T03:51:54.886 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:54.886 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 376535, 2024-04-13T03:51:54.886 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:54.886 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:54.886 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 376535, 2024-04-13T03:51:54.887 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 76457877, 2024-04-13T03:51:54.887 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:54.887 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.887 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.887 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.887 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:54.887 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.887 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:54.888 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:54.888 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-13T03:51:54.888 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:54.888 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:54.888 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:54.888 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:54.888 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:54.889 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 35969, 2024-04-13T03:51:54.889 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 156283868, 2024-04-13T03:51:54.889 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 35969, 2024-04-13T03:51:54.889 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 156283868, 2024-04-13T03:51:54.889 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:54.889 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.889 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.890 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.890 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:54.890 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.890 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:54.890 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:54.890 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-13T03:51:54.890 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:54.890 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:54.891 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:54.891 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:54.891 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:54.891 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 35969, 2024-04-13T03:51:54.891 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 35969, 2024-04-13T03:51:54.891 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 35969, 2024-04-13T03:51:54.891 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 35969, 2024-04-13T03:51:54.891 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:54.892 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.892 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.892 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.892 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:54.892 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.892 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:54.892 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:54.892 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-13T03:51:54.893 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:54.893 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 9652, 2024-04-13T03:51:54.893 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 1053484, 2024-04-13T03:51:54.893 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:54.893 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 9652, 2024-04-13T03:51:54.893 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:54.893 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:54.894 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 681, 2024-04-13T03:51:54.894 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 1053484, 2024-04-13T03:51:54.894 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:54.894 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.894 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.894 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.894 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:54.895 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:54.895 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x5608d56e2500": { 2024-04-13T03:51:54.895 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 574, 2024-04-13T03:51:54.895 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-13T03:51:54.895 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:54.895 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 51534, 2024-04-13T03:51:54.895 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 152784252, 2024-04-13T03:51:54.895 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:54.896 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 51534, 2024-04-13T03:51:54.896 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:54.896 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:54.896 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 14097, 2024-04-13T03:51:54.896 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 152783678, 2024-04-13T03:51:54.896 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:54.896 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:54.896 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:54.897 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:54.897 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:54.897 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:54.897 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-13T03:51:55.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:55 smithi188 ceph-mon[36398]: pgmap v539: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 55 op/s 2024-04-13T03:51:55.322 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:55 smithi119 ceph-mon[35835]: pgmap v539: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 55 op/s 2024-04-13T03:51:55.353 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:55 smithi016 ceph-mon[31772]: pgmap v539: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 55 op/s 2024-04-13T03:51:55.649 INFO:tasks.check_counter:Daemon mds.i mds.exported=7 2024-04-13T03:51:55.649 INFO:tasks.check_counter:Daemon mds.i mds.imported=7 2024-04-13T03:51:55.649 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=735 2024-04-13T03:51:55.649 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=1695 2024-04-13T03:51:55.649 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-13T03:51:55.649 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-13T03:51:55.649 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-13T03:51:55.650 DEBUG:tasks.check_counter:Getting stats from l 2024-04-13T03:51:55.650 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 shell --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-13T03:51:55.892 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/mon.c/config 2024-04-13T03:51:57.301 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:57 smithi188 ceph-mon[36398]: pgmap v540: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.2 MiB/s wr, 45 op/s 2024-04-13T03:51:57.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:57 smithi119 ceph-mon[35835]: pgmap v540: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.2 MiB/s wr, 45 op/s 2024-04-13T03:51:57.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:57 smithi016 ceph-mon[31772]: pgmap v540: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 1.2 MiB/s wr, 45 op/s 2024-04-13T03:51:57.754 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-13T03:51:57.755 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-0": { 2024-04-13T03:51:57.755 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 1984, 2024-04-13T03:51:57.755 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 1924, 2024-04-13T03:51:57.755 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 27642906, 2024-04-13T03:51:57.755 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 707820, 2024-04-13T03:51:57.756 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 7, 2024-04-13T03:51:57.756 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 3, 2024-04-13T03:51:57.756 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.225519400, 2024-04-13T03:51:57.756 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.062175183, 2024-04-13T03:51:57.756 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.101460367, 2024-04-13T03:51:57.756 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.057718632, 2024-04-13T03:51:57.756 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:57.757 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 1924, 2024-04-13T03:51:57.757 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.082606136, 2024-04-13T03:51:57.757 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000042934 2024-04-13T03:51:57.757 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.757 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:57.757 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.757 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.758 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.758 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 1986096, 2024-04-13T03:51:57.758 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 213648 2024-04-13T03:51:57.758 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.758 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-1": { 2024-04-13T03:51:57.758 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 2062, 2024-04-13T03:51:57.759 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 2318, 2024-04-13T03:51:57.759 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 52911579, 2024-04-13T03:51:57.759 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 1336673, 2024-04-13T03:51:57.759 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 10, 2024-04-13T03:51:57.759 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 7, 2024-04-13T03:51:57.759 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.178607862, 2024-04-13T03:51:57.759 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.052704662, 2024-04-13T03:51:57.759 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.065379582, 2024-04-13T03:51:57.760 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.046501977, 2024-04-13T03:51:57.760 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:57.760 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 2315, 2024-04-13T03:51:57.760 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.095810200, 2024-04-13T03:51:57.760 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000041386 2024-04-13T03:51:57.760 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.760 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:57.760 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.760 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.761 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.761 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-13T03:51:57.761 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 681536 2024-04-13T03:51:57.761 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.762 INFO:teuthology.orchestra.run.smithi188.stdout: "AsyncMessenger::Worker-2": { 2024-04-13T03:51:57.762 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_messages": 448, 2024-04-13T03:51:57.762 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages": 450, 2024-04-13T03:51:57.762 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_bytes": 50446037, 2024-04-13T03:51:57.762 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_bytes": 142476, 2024-04-13T03:51:57.762 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_created_connections": 7, 2024-04-13T03:51:57.763 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_active_connections": 5, 2024-04-13T03:51:57.763 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_total_time": 0.059180917, 2024-04-13T03:51:57.763 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_send_time": 0.009965562, 2024-04-13T03:51:57.763 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_recv_time": 0.035750769, 2024-04-13T03:51:57.763 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_running_fast_dispatch_time": 0.010450208, 2024-04-13T03:51:57.763 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_messages_queue_lat": { 2024-04-13T03:51:57.763 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 448, 2024-04-13T03:51:57.764 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.017421703, 2024-04-13T03:51:57.764 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000038887 2024-04-13T03:51:57.764 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.764 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_handle_ack_lat": { 2024-04-13T03:51:57.764 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.764 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.764 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.765 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-13T03:51:57.765 INFO:teuthology.orchestra.run.smithi188.stdout: "msgr_send_encrypted_bytes": 0 2024-04-13T03:51:57.765 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.765 INFO:teuthology.orchestra.run.smithi188.stdout: "cct": { 2024-04-13T03:51:57.766 INFO:teuthology.orchestra.run.smithi188.stdout: "total_workers": 1, 2024-04-13T03:51:57.766 INFO:teuthology.orchestra.run.smithi188.stdout: "unhealthy_workers": 0 2024-04-13T03:51:57.766 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.766 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-MDSRank": { 2024-04-13T03:51:57.766 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-13T03:51:57.766 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-13T03:51:57.767 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 6982, 2024-04-13T03:51:57.767 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.159895603, 2024-04-13T03:51:57.767 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000022901 2024-04-13T03:51:57.767 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:57.767 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.768 INFO:teuthology.orchestra.run.smithi188.stdout: "finisher-PurgeQueue": { 2024-04-13T03:51:57.768 INFO:teuthology.orchestra.run.smithi188.stdout: "queue_len": 0, 2024-04-13T03:51:57.768 INFO:teuthology.orchestra.run.smithi188.stdout: "complete_latency": { 2024-04-13T03:51:57.768 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.768 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.768 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.769 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:57.769 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.769 INFO:teuthology.orchestra.run.smithi188.stdout: "mds": { 2024-04-13T03:51:57.769 INFO:teuthology.orchestra.run.smithi188.stdout: "request": 0, 2024-04-13T03:51:57.769 INFO:teuthology.orchestra.run.smithi188.stdout: "reply": 0, 2024-04-13T03:51:57.769 INFO:teuthology.orchestra.run.smithi188.stdout: "reply_latency": { 2024-04-13T03:51:57.769 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.769 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.770 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.770 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.770 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_reply": 0, 2024-04-13T03:51:57.770 INFO:teuthology.orchestra.run.smithi188.stdout: "forward": 0, 2024-04-13T03:51:57.771 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_complete": 0, 2024-04-13T03:51:57.771 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_fetch_keys": 0, 2024-04-13T03:51:57.771 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_commit": 0, 2024-04-13T03:51:57.771 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_split": 0, 2024-04-13T03:51:57.771 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_merge": 0, 2024-04-13T03:51:57.771 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes": 7731, 2024-04-13T03:51:57.771 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_top": 2, 2024-04-13T03:51:57.772 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_bottom": 7729, 2024-04-13T03:51:57.772 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pin_tail": 0, 2024-04-13T03:51:57.772 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_pinned": 7729, 2024-04-13T03:51:57.772 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_expired": 0, 2024-04-13T03:51:57.772 INFO:teuthology.orchestra.run.smithi188.stdout: "inodes_with_caps": 0, 2024-04-13T03:51:57.772 INFO:teuthology.orchestra.run.smithi188.stdout: "caps": 0, 2024-04-13T03:51:57.772 INFO:teuthology.orchestra.run.smithi188.stdout: "subtrees": 6, 2024-04-13T03:51:57.772 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse": 0, 2024-04-13T03:51:57.772 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_hit": 0, 2024-04-13T03:51:57.773 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_forward": 0, 2024-04-13T03:51:57.773 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_discover": 0, 2024-04-13T03:51:57.773 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_dir_fetch": 0, 2024-04-13T03:51:57.773 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_remote_ino": 0, 2024-04-13T03:51:57.773 INFO:teuthology.orchestra.run.smithi188.stdout: "traverse_lock": 0, 2024-04-13T03:51:57.773 INFO:teuthology.orchestra.run.smithi188.stdout: "load_cent": 0, 2024-04-13T03:51:57.773 INFO:teuthology.orchestra.run.smithi188.stdout: "q": 0, 2024-04-13T03:51:57.773 INFO:teuthology.orchestra.run.smithi188.stdout: "exported": 0, 2024-04-13T03:51:57.774 INFO:teuthology.orchestra.run.smithi188.stdout: "exported_inodes": 0, 2024-04-13T03:51:57.774 INFO:teuthology.orchestra.run.smithi188.stdout: "imported": 0, 2024-04-13T03:51:57.774 INFO:teuthology.orchestra.run.smithi188.stdout: "imported_inodes": 0, 2024-04-13T03:51:57.774 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_dir_fetch": 0, 2024-04-13T03:51:57.774 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_backtrace_fetch": 0, 2024-04-13T03:51:57.774 INFO:teuthology.orchestra.run.smithi188.stdout: "openino_peer_discover": 0, 2024-04-13T03:51:57.774 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rfiles": 0, 2024-04-13T03:51:57.774 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rbytes": 0, 2024-04-13T03:51:57.774 INFO:teuthology.orchestra.run.smithi188.stdout: "root_rsnaps": 0, 2024-04-13T03:51:57.775 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_fetch": 0, 2024-04-13T03:51:57.775 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_set_tag": 0, 2024-04-13T03:51:57.775 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_backtrace_repaired": 0, 2024-04-13T03:51:57.775 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_inotable_repaired": 0, 2024-04-13T03:51:57.775 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_inodes": 0, 2024-04-13T03:51:57.775 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dir_base_inodes": 0, 2024-04-13T03:51:57.775 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_dirfrag_rstats": 0, 2024-04-13T03:51:57.775 INFO:teuthology.orchestra.run.smithi188.stdout: "scrub_file_inodes": 0, 2024-04-13T03:51:57.775 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_inode_file_caps": 0, 2024-04-13T03:51:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_revoke": 0, 2024-04-13T03:51:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_grant": 0, 2024-04-13T03:51:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_trunc": 0, 2024-04-13T03:51:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-13T03:51:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-13T03:51:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps": 0, 2024-04-13T03:51:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_caps_dirty": 0, 2024-04-13T03:51:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_cap_release": 0, 2024-04-13T03:51:57.776 INFO:teuthology.orchestra.run.smithi188.stdout: "process_request_cap_release": 0 2024-04-13T03:51:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_cache": { 2024-04-13T03:51:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update": 0, 2024-04-13T03:51:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_update_receipt": 0, 2024-04-13T03:51:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_try_discover": 0, 2024-04-13T03:51:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_send_discover": 0, 2024-04-13T03:51:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: "dir_handle_discover": 0, 2024-04-13T03:51:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays": 0, 2024-04-13T03:51:57.777 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_delayed": 0, 2024-04-13T03:51:57.778 INFO:teuthology.orchestra.run.smithi188.stdout: "num_strays_enqueuing": 0, 2024-04-13T03:51:57.778 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_created": 0, 2024-04-13T03:51:57.778 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_enqueued": 0, 2024-04-13T03:51:57.778 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_reintegrated": 0, 2024-04-13T03:51:57.778 INFO:teuthology.orchestra.run.smithi188.stdout: "strays_migrated": 0, 2024-04-13T03:51:57.778 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_processing": 0, 2024-04-13T03:51:57.778 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_enqueued": 0, 2024-04-13T03:51:57.778 INFO:teuthology.orchestra.run.smithi188.stdout: "num_recovering_prioritized": 0, 2024-04-13T03:51:57.779 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_started": 0, 2024-04-13T03:51:57.779 INFO:teuthology.orchestra.run.smithi188.stdout: "recovery_completed": 0, 2024-04-13T03:51:57.779 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_path": 0, 2024-04-13T03:51:57.779 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_quiesce_inode": 0, 2024-04-13T03:51:57.779 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_enqueue_scrub": 0, 2024-04-13T03:51:57.779 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_exportdir": 0, 2024-04-13T03:51:57.779 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_flush": 0, 2024-04-13T03:51:57.779 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragmentdir": 0, 2024-04-13T03:51:57.779 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_fragstats": 0, 2024-04-13T03:51:57.781 INFO:teuthology.orchestra.run.smithi188.stdout: "ireq_inodestats": 0 2024-04-13T03:51:57.781 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.781 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_log": { 2024-04-13T03:51:57.781 INFO:teuthology.orchestra.run.smithi188.stdout: "evlrg": 0, 2024-04-13T03:51:57.781 INFO:teuthology.orchestra.run.smithi188.stdout: "evadd": 0, 2024-04-13T03:51:57.781 INFO:teuthology.orchestra.run.smithi188.stdout: "evex": 0, 2024-04-13T03:51:57.782 INFO:teuthology.orchestra.run.smithi188.stdout: "evtrm": 0, 2024-04-13T03:51:57.782 INFO:teuthology.orchestra.run.smithi188.stdout: "ev": 43570, 2024-04-13T03:51:57.782 INFO:teuthology.orchestra.run.smithi188.stdout: "evexg": 0, 2024-04-13T03:51:57.782 INFO:teuthology.orchestra.run.smithi188.stdout: "evexd": 0, 2024-04-13T03:51:57.782 INFO:teuthology.orchestra.run.smithi188.stdout: "segadd": 0, 2024-04-13T03:51:57.782 INFO:teuthology.orchestra.run.smithi188.stdout: "segex": 0, 2024-04-13T03:51:57.782 INFO:teuthology.orchestra.run.smithi188.stdout: "segtrm": 0, 2024-04-13T03:51:57.782 INFO:teuthology.orchestra.run.smithi188.stdout: "seg": 62, 2024-04-13T03:51:57.783 INFO:teuthology.orchestra.run.smithi188.stdout: "segmjr": 5, 2024-04-13T03:51:57.783 INFO:teuthology.orchestra.run.smithi188.stdout: "segexg": 0, 2024-04-13T03:51:57.783 INFO:teuthology.orchestra.run.smithi188.stdout: "segexd": 0, 2024-04-13T03:51:57.783 INFO:teuthology.orchestra.run.smithi188.stdout: "expos": 4194304, 2024-04-13T03:51:57.783 INFO:teuthology.orchestra.run.smithi188.stdout: "wrpos": 132057145, 2024-04-13T03:51:57.783 INFO:teuthology.orchestra.run.smithi188.stdout: "rdpos": 132051729, 2024-04-13T03:51:57.783 INFO:teuthology.orchestra.run.smithi188.stdout: "jlat": { 2024-04-13T03:51:57.783 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.783 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.784 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.784 INFO:teuthology.orchestra.run.smithi188.stdout: "replayed": 43570 2024-04-13T03:51:57.784 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.784 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_mem": { 2024-04-13T03:51:57.784 INFO:teuthology.orchestra.run.smithi188.stdout: "ino": 472, 2024-04-13T03:51:57.784 INFO:teuthology.orchestra.run.smithi188.stdout: "ino+": 7810, 2024-04-13T03:51:57.784 INFO:teuthology.orchestra.run.smithi188.stdout: "ino-": 7338, 2024-04-13T03:51:57.784 INFO:teuthology.orchestra.run.smithi188.stdout: "dir": 41, 2024-04-13T03:51:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "dir+": 83, 2024-04-13T03:51:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "dir-": 42, 2024-04-13T03:51:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "dn": 7747, 2024-04-13T03:51:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "dn+": 7959, 2024-04-13T03:51:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "dn-": 212, 2024-04-13T03:51:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "cap": 0, 2024-04-13T03:51:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "cap+": 0, 2024-04-13T03:51:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "cap-": 0, 2024-04-13T03:51:57.785 INFO:teuthology.orchestra.run.smithi188.stdout: "rss": 62216, 2024-04-13T03:51:57.786 INFO:teuthology.orchestra.run.smithi188.stdout: "heap": 215416 2024-04-13T03:51:57.786 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.786 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_server": { 2024-04-13T03:51:57.786 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_client_request": 0, 2024-04-13T03:51:57.786 INFO:teuthology.orchestra.run.smithi188.stdout: "dispatch_server_request": 0, 2024-04-13T03:51:57.786 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_request": 0, 2024-04-13T03:51:57.786 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_client_session": 0, 2024-04-13T03:51:57.786 INFO:teuthology.orchestra.run.smithi188.stdout: "handle_peer_request": 0, 2024-04-13T03:51:57.786 INFO:teuthology.orchestra.run.smithi188.stdout: "req_create_latency": { 2024-04-13T03:51:57.787 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.787 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.787 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.787 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getattr_latency": { 2024-04-13T03:51:57.787 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.787 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.787 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.787 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.788 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getfilelock_latency": { 2024-04-13T03:51:57.788 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.788 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.788 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.788 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.788 INFO:teuthology.orchestra.run.smithi188.stdout: "req_link_latency": { 2024-04-13T03:51:57.788 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.788 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.788 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.789 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookup_latency": { 2024-04-13T03:51:57.789 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.789 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.789 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.789 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.789 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookuphash_latency": { 2024-04-13T03:51:57.789 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.789 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.790 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.790 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupino_latency": { 2024-04-13T03:51:57.790 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.790 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.790 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.790 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.790 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupname_latency": { 2024-04-13T03:51:57.790 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.791 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.791 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.791 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupparent_latency": { 2024-04-13T03:51:57.791 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.791 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.791 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.791 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.791 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lookupsnap_latency": { 2024-04-13T03:51:57.792 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.792 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.792 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.792 INFO:teuthology.orchestra.run.smithi188.stdout: "req_lssnap_latency": { 2024-04-13T03:51:57.792 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.792 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.792 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.792 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.793 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mkdir_latency": { 2024-04-13T03:51:57.793 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.794 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.794 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.794 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mknod_latency": { 2024-04-13T03:51:57.794 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.794 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.794 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.794 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.795 INFO:teuthology.orchestra.run.smithi188.stdout: "req_mksnap_latency": { 2024-04-13T03:51:57.795 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.795 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.795 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.795 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.795 INFO:teuthology.orchestra.run.smithi188.stdout: "req_open_latency": { 2024-04-13T03:51:57.795 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.795 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.795 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.796 INFO:teuthology.orchestra.run.smithi188.stdout: "req_readdir_latency": { 2024-04-13T03:51:57.796 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.796 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.796 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.796 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.796 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rename_latency": { 2024-04-13T03:51:57.796 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.796 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.797 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.797 INFO:teuthology.orchestra.run.smithi188.stdout: "req_renamesnap_latency": { 2024-04-13T03:51:57.797 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.797 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.797 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.797 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.797 INFO:teuthology.orchestra.run.smithi188.stdout: "req_snapdiff_latency": { 2024-04-13T03:51:57.797 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.798 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.798 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.798 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.798 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmdir_latency": { 2024-04-13T03:51:57.798 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.798 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.798 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.798 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.799 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmsnap_latency": { 2024-04-13T03:51:57.799 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.799 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.799 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.799 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.799 INFO:teuthology.orchestra.run.smithi188.stdout: "req_rmxattr_latency": { 2024-04-13T03:51:57.799 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.799 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.799 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.800 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setattr_latency": { 2024-04-13T03:51:57.800 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.800 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.800 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.800 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.800 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setdirlayout_latency": { 2024-04-13T03:51:57.800 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.800 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.801 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.801 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setfilelock_latency": { 2024-04-13T03:51:57.801 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.801 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.801 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.801 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.801 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setlayout_latency": { 2024-04-13T03:51:57.801 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.802 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.802 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.802 INFO:teuthology.orchestra.run.smithi188.stdout: "req_setxattr_latency": { 2024-04-13T03:51:57.802 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.802 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.802 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.802 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.802 INFO:teuthology.orchestra.run.smithi188.stdout: "req_symlink_latency": { 2024-04-13T03:51:57.803 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.803 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.803 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.803 INFO:teuthology.orchestra.run.smithi188.stdout: "req_unlink_latency": { 2024-04-13T03:51:57.803 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.803 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.803 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.803 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_revoke_eviction": 0, 2024-04-13T03:51:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: "cap_acquisition_throttle": 0, 2024-04-13T03:51:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: "req_getvxattr_latency": { 2024-04-13T03:51:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.804 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_sessions": { 2024-04-13T03:51:57.805 INFO:teuthology.orchestra.run.smithi188.stdout: "session_count": 1, 2024-04-13T03:51:57.805 INFO:teuthology.orchestra.run.smithi188.stdout: "session_add": 3, 2024-04-13T03:51:57.805 INFO:teuthology.orchestra.run.smithi188.stdout: "session_remove": 2, 2024-04-13T03:51:57.805 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_open": 1, 2024-04-13T03:51:57.805 INFO:teuthology.orchestra.run.smithi188.stdout: "sessions_stale": 0, 2024-04-13T03:51:57.805 INFO:teuthology.orchestra.run.smithi188.stdout: "total_load": 0, 2024-04-13T03:51:57.805 INFO:teuthology.orchestra.run.smithi188.stdout: "average_load": 0, 2024-04-13T03:51:57.805 INFO:teuthology.orchestra.run.smithi188.stdout: "avg_session_uptime": 2647, 2024-04-13T03:51:57.805 INFO:teuthology.orchestra.run.smithi188.stdout: "mdthresh_evicted": 0 2024-04-13T03:51:57.806 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.806 INFO:teuthology.orchestra.run.smithi188.stdout: "mempool": { 2024-04-13T03:51:57.806 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_bytes": 0, 2024-04-13T03:51:57.806 INFO:teuthology.orchestra.run.smithi188.stdout: "bloom_filter_items": 0, 2024-04-13T03:51:57.806 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_bytes": 0, 2024-04-13T03:51:57.806 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_alloc_items": 0, 2024-04-13T03:51:57.806 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_bytes": 0, 2024-04-13T03:51:57.806 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_data_items": 0, 2024-04-13T03:51:57.806 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-13T03:51:57.807 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_onode_items": 0, 2024-04-13T03:51:57.807 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-13T03:51:57.807 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_meta_items": 0, 2024-04-13T03:51:57.807 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_bytes": 0, 2024-04-13T03:51:57.807 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_other_items": 0, 2024-04-13T03:51:57.807 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-13T03:51:57.807 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_cache_buffer_items": 0, 2024-04-13T03:51:57.807 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_bytes": 0, 2024-04-13T03:51:57.808 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_extent_items": 0, 2024-04-13T03:51:57.808 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_bytes": 0, 2024-04-13T03:51:57.808 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_blob_items": 0, 2024-04-13T03:51:57.808 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-13T03:51:57.808 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_shared_blob_items": 0, 2024-04-13T03:51:57.808 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-13T03:51:57.808 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_inline_bl_items": 0, 2024-04-13T03:51:57.808 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_bytes": 0, 2024-04-13T03:51:57.808 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_fsck_items": 0, 2024-04-13T03:51:57.809 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_bytes": 0, 2024-04-13T03:51:57.809 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_txc_items": 0, 2024-04-13T03:51:57.809 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-13T03:51:57.809 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_deferred_items": 0, 2024-04-13T03:51:57.809 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_bytes": 0, 2024-04-13T03:51:57.809 INFO:teuthology.orchestra.run.smithi188.stdout: "bluestore_writing_items": 0, 2024-04-13T03:51:57.809 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_bytes": 0, 2024-04-13T03:51:57.809 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_items": 0, 2024-04-13T03:51:57.809 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_bytes": 0, 2024-04-13T03:51:57.810 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_reader_items": 0, 2024-04-13T03:51:57.810 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_bytes": 0, 2024-04-13T03:51:57.810 INFO:teuthology.orchestra.run.smithi188.stdout: "bluefs_file_writer_items": 0, 2024-04-13T03:51:57.810 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_bytes": 258473, 2024-04-13T03:51:57.810 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_anon_items": 594, 2024-04-13T03:51:57.810 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_bytes": 0, 2024-04-13T03:51:57.810 INFO:teuthology.orchestra.run.smithi188.stdout: "buffer_meta_items": 0, 2024-04-13T03:51:57.810 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_bytes": 0, 2024-04-13T03:51:57.810 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_items": 0, 2024-04-13T03:51:57.811 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_bytes": 0, 2024-04-13T03:51:57.811 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_mapbl_items": 0, 2024-04-13T03:51:57.811 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_bytes": 0, 2024-04-13T03:51:57.811 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_pglog_items": 0, 2024-04-13T03:51:57.811 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_bytes": 9704, 2024-04-13T03:51:57.811 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_items": 186, 2024-04-13T03:51:57.811 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_bytes": 0, 2024-04-13T03:51:57.811 INFO:teuthology.orchestra.run.smithi188.stdout: "osdmap_mapping_items": 0, 2024-04-13T03:51:57.811 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_bytes": 0, 2024-04-13T03:51:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: "pgmap_items": 0, 2024-04-13T03:51:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_bytes": 6709417, 2024-04-13T03:51:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_co_items": 43313, 2024-04-13T03:51:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_bytes": 0, 2024-04-13T03:51:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_1_items": 0, 2024-04-13T03:51:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_bytes": 0, 2024-04-13T03:51:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: "unittest_2_items": 0 2024-04-13T03:51:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.812 INFO:teuthology.orchestra.run.smithi188.stdout: "objecter": { 2024-04-13T03:51:57.813 INFO:teuthology.orchestra.run.smithi188.stdout: "op_active": 0, 2024-04-13T03:51:57.813 INFO:teuthology.orchestra.run.smithi188.stdout: "op_laggy": 0, 2024-04-13T03:51:57.813 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send": 4062, 2024-04-13T03:51:57.814 INFO:teuthology.orchestra.run.smithi188.stdout: "op_send_bytes": 21, 2024-04-13T03:51:57.814 INFO:teuthology.orchestra.run.smithi188.stdout: "op_resend": 3, 2024-04-13T03:51:57.814 INFO:teuthology.orchestra.run.smithi188.stdout: "op_reply": 4062, 2024-04-13T03:51:57.814 INFO:teuthology.orchestra.run.smithi188.stdout: "op_latency": { 2024-04-13T03:51:57.814 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 4062, 2024-04-13T03:51:57.814 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 6.153886457, 2024-04-13T03:51:57.814 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.001514989 2024-04-13T03:51:57.814 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: "op_inflight": 0, 2024-04-13T03:51:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: "oplen_avg": { 2024-04-13T03:51:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 4062, 2024-04-13T03:51:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 4063 2024-04-13T03:51:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: "op": 4062, 2024-04-13T03:51:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: "op_r": 4062, 2024-04-13T03:51:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: "op_w": 0, 2024-04-13T03:51:57.815 INFO:teuthology.orchestra.run.smithi188.stdout: "op_rmw": 0, 2024-04-13T03:51:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: "op_pg": 0, 2024-04-13T03:51:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_stat": 2269, 2024-04-13T03:51:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_create": 0, 2024-04-13T03:51:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_read": 1791, 2024-04-13T03:51:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_write": 0, 2024-04-13T03:51:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writefull": 0, 2024-04-13T03:51:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_writesame": 0, 2024-04-13T03:51:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_append": 0, 2024-04-13T03:51:57.816 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_zero": 0, 2024-04-13T03:51:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_truncate": 0, 2024-04-13T03:51:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_delete": 0, 2024-04-13T03:51:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_mapext": 0, 2024-04-13T03:51:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_sparse_read": 0, 2024-04-13T03:51:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_clonerange": 0, 2024-04-13T03:51:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_getxattr": 1, 2024-04-13T03:51:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_setxattr": 0, 2024-04-13T03:51:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_cmpxattr": 0, 2024-04-13T03:51:57.817 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_rmxattr": 0, 2024-04-13T03:51:57.818 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_resetxattrs": 0, 2024-04-13T03:51:57.818 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_call": 0, 2024-04-13T03:51:57.818 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_watch": 0, 2024-04-13T03:51:57.818 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_notify": 0, 2024-04-13T03:51:57.818 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_src_cmpxattr": 0, 2024-04-13T03:51:57.818 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls": 0, 2024-04-13T03:51:57.818 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_pgls_filter": 0, 2024-04-13T03:51:57.818 INFO:teuthology.orchestra.run.smithi188.stdout: "osdop_other": 0, 2024-04-13T03:51:57.818 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_active": 0, 2024-04-13T03:51:57.819 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_send": 0, 2024-04-13T03:51:57.819 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_resend": 0, 2024-04-13T03:51:57.819 INFO:teuthology.orchestra.run.smithi188.stdout: "linger_ping": 0, 2024-04-13T03:51:57.819 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_active": 0, 2024-04-13T03:51:57.819 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_send": 0, 2024-04-13T03:51:57.819 INFO:teuthology.orchestra.run.smithi188.stdout: "poolop_resend": 0, 2024-04-13T03:51:57.819 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_active": 0, 2024-04-13T03:51:57.819 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_send": 0, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "poolstat_resend": 0, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_active": 0, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_send": 0, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "statfs_resend": 0, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "command_active": 0, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "command_send": 0, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "command_resend": 0, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "map_epoch": 117, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "map_full": 0, 2024-04-13T03:51:57.820 INFO:teuthology.orchestra.run.smithi188.stdout: "map_inc": 27, 2024-04-13T03:51:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_sessions": 11, 2024-04-13T03:51:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_open": 12, 2024-04-13T03:51:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_session_close": 1, 2024-04-13T03:51:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_laggy": 0, 2024-04-13T03:51:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_wr": 0, 2024-04-13T03:51:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_rd": 2, 2024-04-13T03:51:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_del": 0 2024-04-13T03:51:57.821 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.822 INFO:teuthology.orchestra.run.smithi188.stdout: "oft": { 2024-04-13T03:51:57.822 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_objs": 0, 2024-04-13T03:51:57.822 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_kv_pairs": 0, 2024-04-13T03:51:57.822 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_updates": 0, 2024-04-13T03:51:57.822 INFO:teuthology.orchestra.run.smithi188.stdout: "omap_total_removes": 0 2024-04-13T03:51:57.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.822 INFO:teuthology.orchestra.run.smithi188.stdout: "osd-slow-ops": { 2024-04-13T03:51:57.822 INFO:teuthology.orchestra.run.smithi188.stdout: "slow_ops_count": 0 2024-04-13T03:51:57.822 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.823 INFO:teuthology.orchestra.run.smithi188.stdout: "purge_queue": { 2024-04-13T03:51:57.823 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops": 0, 2024-04-13T03:51:57.823 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_ops_high_water": 0, 2024-04-13T03:51:57.823 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing": 0, 2024-04-13T03:51:57.823 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executing_high_water": 0, 2024-04-13T03:51:57.823 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed_ops": 0, 2024-04-13T03:51:57.823 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_executed": 0, 2024-04-13T03:51:57.823 INFO:teuthology.orchestra.run.smithi188.stdout: "pq_item_in_journal": 0 2024-04-13T03:51:57.823 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.824 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-13T03:51:57.824 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:57.824 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-13T03:51:57.824 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:57.824 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 4490, 2024-04-13T03:51:57.824 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 130571746, 2024-04-13T03:51:57.824 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:57.824 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 4490, 2024-04-13T03:51:57.824 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:57.825 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:57.825 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 4490, 2024-04-13T03:51:57.825 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 130571746, 2024-04-13T03:51:57.825 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:57.825 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.825 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.825 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.825 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:57.825 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.826 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_bytes": { 2024-04-13T03:51:57.826 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:57.826 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 104857600, 2024-04-13T03:51:57.826 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:57.826 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:57.826 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:57.826 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:57.826 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:57.826 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 4062, 2024-04-13T03:51:57.827 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 127885624, 2024-04-13T03:51:57.827 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 4062, 2024-04-13T03:51:57.827 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 127885624, 2024-04-13T03:51:57.827 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:57.827 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.827 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.827 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.827 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:57.827 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-objecter_ops": { 2024-04-13T03:51:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 1024, 2024-04-13T03:51:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:57.828 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 4062, 2024-04-13T03:51:57.829 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 4062, 2024-04-13T03:51:57.829 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 4062, 2024-04-13T03:51:57.829 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 4062, 2024-04-13T03:51:57.829 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:57.829 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.829 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.829 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.829 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:57.829 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.830 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle": { 2024-04-13T03:51:57.830 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:57.830 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-13T03:51:57.830 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:57.830 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:57.830 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:57.830 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:57.830 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:57.831 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:57.831 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:57.831 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2024-04-13T03:51:57.831 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2024-04-13T03:51:57.831 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:57.831 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.831 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.831 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.831 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:57.832 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-13T03:51:57.832 INFO:teuthology.orchestra.run.smithi188.stdout: "throttle-write_buf_throttle-0x5604725206e0": { 2024-04-13T03:51:57.832 INFO:teuthology.orchestra.run.smithi188.stdout: "val": 0, 2024-04-13T03:51:57.832 INFO:teuthology.orchestra.run.smithi188.stdout: "max": 3758096384, 2024-04-13T03:51:57.832 INFO:teuthology.orchestra.run.smithi188.stdout: "get_started": 0, 2024-04-13T03:51:57.832 INFO:teuthology.orchestra.run.smithi188.stdout: "get": 0, 2024-04-13T03:51:57.832 INFO:teuthology.orchestra.run.smithi188.stdout: "get_sum": 0, 2024-04-13T03:51:57.832 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_fail": 0, 2024-04-13T03:51:57.832 INFO:teuthology.orchestra.run.smithi188.stdout: "get_or_fail_success": 0, 2024-04-13T03:51:57.833 INFO:teuthology.orchestra.run.smithi188.stdout: "take": 0, 2024-04-13T03:51:57.833 INFO:teuthology.orchestra.run.smithi188.stdout: "take_sum": 0, 2024-04-13T03:51:57.833 INFO:teuthology.orchestra.run.smithi188.stdout: "put": 0, 2024-04-13T03:51:57.833 INFO:teuthology.orchestra.run.smithi188.stdout: "put_sum": 0, 2024-04-13T03:51:57.833 INFO:teuthology.orchestra.run.smithi188.stdout: "wait": { 2024-04-13T03:51:57.833 INFO:teuthology.orchestra.run.smithi188.stdout: "avgcount": 0, 2024-04-13T03:51:57.833 INFO:teuthology.orchestra.run.smithi188.stdout: "sum": 0.000000000, 2024-04-13T03:51:57.833 INFO:teuthology.orchestra.run.smithi188.stdout: "avgtime": 0.000000000 2024-04-13T03:51:57.833 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:57.834 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-13T03:51:57.834 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-13T03:51:58.320 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:58 smithi119 ceph-mon[35835]: pgmap v541: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-04-13T03:51:58.354 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:58 smithi016 ceph-mon[31772]: pgmap v541: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-04-13T03:51:58.372 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:58 smithi188 ceph-mon[36398]: pgmap v541: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-04-13T03:51:58.654 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-13T03:51:58.655 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-13T03:51:58.655 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-04-13T03:51:58.655 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-04-13T03:51:58.655 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-04-13T03:51:58.655 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-13T03:51:58.655 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-13T03:51:58.656 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-13T03:51:58.668 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-13T03:51:58.668 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:51:58.668 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-13T03:51:58.732 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-13T03:51:58.732 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-13T03:51:58.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:51:59.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.035+0000 7fc2b2200640 1 -- 172.21.15.16:0/164223780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2ac10c3a0 msgr2=0x7fc2ac10e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.035+0000 7fc2b2200640 1 --2- 172.21.15.16:0/164223780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2ac10c3a0 0x7fc2ac10e820 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009be0 tx=0x7fc2a402bf80 comp rx=0 tx=0).stop 2024-04-13T03:51:59.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.035+0000 7fc2b2200640 1 -- 172.21.15.16:0/164223780 shutdown_connections 2024-04-13T03:51:59.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.035+0000 7fc2b2200640 1 --2- 172.21.15.16:0/164223780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2ac10c3a0 0x7fc2ac10e820 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.035+0000 7fc2b2200640 1 --2- 172.21.15.16:0/164223780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2ac1038f0 0x7fc2ac10bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.035+0000 7fc2b2200640 1 --2- 172.21.15.16:0/164223780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2ac102fd0 0x7fc2ac1033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.035+0000 7fc2b2200640 1 -- 172.21.15.16:0/164223780 >> 172.21.15.16:0/164223780 conn(0x7fc2ac0fc790 msgr2=0x7fc2ac0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:59.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.036+0000 7fc2b2200640 1 -- 172.21.15.16:0/164223780 shutdown_connections 2024-04-13T03:51:59.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.036+0000 7fc2b2200640 1 -- 172.21.15.16:0/164223780 wait complete. 2024-04-13T03:51:59.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b2200640 1 Processor -- start 2024-04-13T03:51:59.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b2200640 1 -- start start 2024-04-13T03:51:59.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b2200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2ac102fd0 0x7fc2ac1a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b2200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2ac1038f0 0x7fc2ac1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b2200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2ac10c3a0 0x7fc2ac19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b2200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ac112190 con 0x7fc2ac102fd0 2024-04-13T03:51:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b2200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ac112010 con 0x7fc2ac10c3a0 2024-04-13T03:51:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b2200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ac112310 con 0x7fc2ac1038f0 2024-04-13T03:51:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2ac102fd0 0x7fc2ac1a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b0e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2ac102fd0 0x7fc2ac1a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40688/0 (socket says 172.21.15.16:40688) 2024-04-13T03:51:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b0e00640 1 -- 172.21.15.16:0/3003338256 learned_addr learned my addr 172.21.15.16:0/3003338256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2b1800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2ac10c3a0 0x7fc2ac19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.037+0000 7fc2abe00640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2ac1038f0 0x7fc2ac1a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.038+0000 7fc2b0e00640 1 -- 172.21.15.16:0/3003338256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2ac1038f0 msgr2=0x7fc2ac1a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.038+0000 7fc2b0e00640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2ac1038f0 0x7fc2ac1a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.038+0000 7fc2b0e00640 1 -- 172.21.15.16:0/3003338256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2ac10c3a0 msgr2=0x7fc2ac19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.038+0000 7fc2b0e00640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2ac10c3a0 0x7fc2ac19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.038+0000 7fc2b0e00640 1 -- 172.21.15.16:0/3003338256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ac19ada0 con 0x7fc2ac102fd0 2024-04-13T03:51:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.038+0000 7fc2b0e00640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2ac102fd0 0x7fc2ac1a0470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc2a000d950 tx=0x7fc2a000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.038+0000 7fc2a9600640 1 -- 172.21.15.16:0/3003338256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0017030 con 0x7fc2ac102fd0 2024-04-13T03:51:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.039+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2ac19b090 con 0x7fc2ac102fd0 2024-04-13T03:51:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.039+0000 7fc2a9600640 1 -- 172.21.15.16:0/3003338256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2a0004020 con 0x7fc2ac102fd0 2024-04-13T03:51:59.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.039+0000 7fc2a9600640 1 -- 172.21.15.16:0/3003338256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0005650 con 0x7fc2ac102fd0 2024-04-13T03:51:59.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.039+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2ac1a7330 con 0x7fc2ac102fd0 2024-04-13T03:51:59.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.040+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2ac069150 con 0x7fc2ac102fd0 2024-04-13T03:51:59.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.042+0000 7fc2a9600640 1 -- 172.21.15.16:0/3003338256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc2a00041d0 con 0x7fc2ac102fd0 2024-04-13T03:51:59.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.044+0000 7fc2a9600640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc2940a3d20 0x7fc2940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.044+0000 7fc2a9600640 1 -- 172.21.15.16:0/3003338256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fc2a00d0bf0 con 0x7fc2ac102fd0 2024-04-13T03:51:59.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.044+0000 7fc2abe00640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc2940a3d20 0x7fc2940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.045+0000 7fc2abe00640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc2940a3d20 0x7fc2940a61e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac10ecf0 tx=0x7fc298009260 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:59.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.047+0000 7fc2a9600640 1 -- 172.21.15.16:0/3003338256 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2a009ce80 con 0x7fc2ac102fd0 2024-04-13T03:51:59.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.256+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2ac19bb50 con 0x7fc2ac102fd0 2024-04-13T03:51:59.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.259+0000 7fc2a9600640 1 -- 172.21.15.16:0/3003338256 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+25440 (secure 0 0 0) 0x7fc2a009c780 con 0x7fc2ac102fd0 2024-04-13T03:51:59.260 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:51:59.260 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:51:08.244014+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2756805961":"2024-04-14T03:34:45.365818+0000","172.21.15.16:0/2348781121":"2024-04-14T03:34:45.365818+0000","172.21.15.16:0/1183476525":"2024-04-14T03:34:45.365818+0000","172.21.15.16:6800/1721562410":"2024-04-14T03:34:45.365818+0000","172.21.15.16:0/2242726589":"2024-04-14T03:34:45.365818+0000","172.21.15.16:6801/1721562410":"2024-04-14T03:34:45.365818+0000","172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]},{"pool":4,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:51:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc2940a3d20 msgr2=0x7fc2940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc2940a3d20 0x7fc2940a61e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac10ecf0 tx=0x7fc298009260 comp rx=0 tx=0).stop 2024-04-13T03:51:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2ac102fd0 msgr2=0x7fc2ac1a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2ac102fd0 0x7fc2ac1a0470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc2a000d950 tx=0x7fc2a000dd50 comp rx=0 tx=0).stop 2024-04-13T03:51:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 shutdown_connections 2024-04-13T03:51:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fc2940a3d20 0x7fc2940a61e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2ac10c3a0 0x7fc2ac19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2ac1038f0 0x7fc2ac1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 --2- 172.21.15.16:0/3003338256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc2ac102fd0 0x7fc2ac1a0470 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.262+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 >> 172.21.15.16:0/3003338256 conn(0x7fc2ac0fc790 msgr2=0x7fc2ac100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.263+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 shutdown_connections 2024-04-13T03:51:59.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.263+0000 7fc2b2200640 1 -- 172.21.15.16:0/3003338256 wait complete. 2024-04-13T03:51:59.283 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-13T03:26:21.881220+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-13T03:31:34.802688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-13T03:31:36.036786+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 116, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-13T03:31:39.824413+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 116, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-13T03:51:59.284 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-13T03:51:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.511+0000 7f28d3e00640 1 -- 172.21.15.16:0/2841943981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4074c80 msgr2=0x7f28d4112970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.511+0000 7f28d3e00640 1 --2- 172.21.15.16:0/2841943981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4074c80 0x7f28d4112970 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009900 tx=0x7f28bc02bcd0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.511+0000 7f28d3e00640 1 -- 172.21.15.16:0/2841943981 shutdown_connections 2024-04-13T03:51:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.511+0000 7f28d3e00640 1 --2- 172.21.15.16:0/2841943981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28d4112eb0 0x7f28d41152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.511+0000 7f28d3e00640 1 --2- 172.21.15.16:0/2841943981 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4074c80 0x7f28d4112970 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.511+0000 7f28d3e00640 1 --2- 172.21.15.16:0/2841943981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28d4074360 0x7f28d4074740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.511+0000 7f28d3e00640 1 -- 172.21.15.16:0/2841943981 >> 172.21.15.16:0/2841943981 conn(0x7f28d40fd9d0 msgr2=0x7f28d40ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.512+0000 7f28d3e00640 1 -- 172.21.15.16:0/2841943981 shutdown_connections 2024-04-13T03:51:59.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.512+0000 7f28d3e00640 1 -- 172.21.15.16:0/2841943981 wait complete. 2024-04-13T03:51:59.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3e00640 1 Processor -- start 2024-04-13T03:51:59.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3e00640 1 -- start start 2024-04-13T03:51:59.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28d4074360 0x7f28d419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28d4074c80 0x7f28d419f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4112eb0 0x7f28d41a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28d41188f0 con 0x7f28d4112eb0 2024-04-13T03:51:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f28d4118770 con 0x7f28d4074360 2024-04-13T03:51:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f28d4118a70 con 0x7f28d4074c80 2024-04-13T03:51:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4112eb0 0x7f28d41a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d2000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28d4074c80 0x7f28d419f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3400640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4112eb0 0x7f28d41a36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40712/0 (socket says 172.21.15.16:40712) 2024-04-13T03:51:59.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.513+0000 7f28d3400640 1 -- 172.21.15.16:0/1387282549 learned_addr learned my addr 172.21.15.16:0/1387282549 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:59.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.514+0000 7f28d2a00640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28d4074360 0x7f28d419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.514+0000 7f28d2000640 1 -- 172.21.15.16:0/1387282549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28d4074360 msgr2=0x7f28d419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.514+0000 7f28d2000640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28d4074360 0x7f28d419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.514+0000 7f28d2000640 1 -- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4112eb0 msgr2=0x7f28d41a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.514+0000 7f28d2000640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4112eb0 0x7f28d41a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.514+0000 7f28d2000640 1 -- 172.21.15.16:0/1387282549 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d41a3e20 con 0x7f28d4074c80 2024-04-13T03:51:59.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.514+0000 7f28d3400640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4112eb0 0x7f28d41a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:51:59.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.514+0000 7f28d2000640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28d4074c80 0x7f28d419f370 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f28bc02c210 tx=0x7f28bc004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:59.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.515+0000 7f28cb400640 1 -- 172.21.15.16:0/1387282549 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc042020 con 0x7f28d4074c80 2024-04-13T03:51:59.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.515+0000 7f28cb400640 1 -- 172.21.15.16:0/1387282549 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28bc0054e0 con 0x7f28d4074c80 2024-04-13T03:51:59.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.515+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d41a40b0 con 0x7f28d4074c80 2024-04-13T03:51:59.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.515+0000 7f28cb400640 1 -- 172.21.15.16:0/1387282549 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc003b00 con 0x7f28d4074c80 2024-04-13T03:51:59.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.515+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d41ab9a0 con 0x7f28d4074c80 2024-04-13T03:51:59.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.516+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28d4068d00 con 0x7f28d4074c80 2024-04-13T03:51:59.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.518+0000 7f28cb400640 1 -- 172.21.15.16:0/1387282549 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f28bc045020 con 0x7f28d4074c80 2024-04-13T03:51:59.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.519+0000 7f28cb400640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f28b80a3da0 0x7f28b80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.519+0000 7f28cb400640 1 -- 172.21.15.16:0/1387282549 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f28bc0f2510 con 0x7f28d4074c80 2024-04-13T03:51:59.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.519+0000 7f28d2a00640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f28b80a3da0 0x7f28b80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.520+0000 7f28d2a00640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f28b80a3da0 0x7f28b80a6260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28c4005fd0 tx=0x7f28c4009500 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:59.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.523+0000 7f28cb400640 1 -- 172.21.15.16:0/1387282549 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28bc04c020 con 0x7f28d4074c80 2024-04-13T03:51:59.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:51:59 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3003338256' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:51:59.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:51:59 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3003338256' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:51:59.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:51:59 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3003338256' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:51:59.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.729+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f28d41abbd0 con 0x7f28d4074c80 2024-04-13T03:51:59.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.730+0000 7f28cb400640 1 -- 172.21.15.16:0/1387282549 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) ==== 93+0+10 (secure 0 0 0) 0x7f28bc0be950 con 0x7f28d4074c80 2024-04-13T03:51:59.729 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-13T03:51:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.732+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f28b80a3da0 msgr2=0x7f28b80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.732+0000 7f28d3e00640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f28b80a3da0 0x7f28b80a6260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28c4005fd0 tx=0x7f28c4009500 comp rx=0 tx=0).stop 2024-04-13T03:51:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28d4074c80 msgr2=0x7f28d419f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28d4074c80 0x7f28d419f370 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f28bc02c210 tx=0x7f28bc004360 comp rx=0 tx=0).stop 2024-04-13T03:51:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 shutdown_connections 2024-04-13T03:51:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f28b80a3da0 0x7f28b80a6260 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28d4112eb0 0x7f28d41a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28d4074c80 0x7f28d419f370 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 --2- 172.21.15.16:0/1387282549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28d4074360 0x7f28d419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 >> 172.21.15.16:0/1387282549 conn(0x7f28d40fd9d0 msgr2=0x7f28d40ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:59.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 shutdown_connections 2024-04-13T03:51:59.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.733+0000 7f28d3e00640 1 -- 172.21.15.16:0/1387282549 wait complete. 2024-04-13T03:51:59.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-13T03:51:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.982+0000 7f6494600640 1 -- 172.21.15.16:0/848151481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f648c104e20 msgr2=0x7f648c108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.982+0000 7f6494600640 1 --2- 172.21.15.16:0/848151481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f648c104e20 0x7f648c108e60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6478009900 tx=0x7f647802bc30 comp rx=0 tx=0).stop 2024-04-13T03:51:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.982+0000 7f6494600640 1 -- 172.21.15.16:0/848151481 shutdown_connections 2024-04-13T03:51:59.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.982+0000 7f6494600640 1 --2- 172.21.15.16:0/848151481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f648c109540 0x7f648c111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.982+0000 7f6494600640 1 --2- 172.21.15.16:0/848151481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f648c104e20 0x7f648c108e60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.982+0000 7f6494600640 1 --2- 172.21.15.16:0/848151481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648c104470 0x7f648c104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.982+0000 7f6494600640 1 -- 172.21.15.16:0/848151481 >> 172.21.15.16:0/848151481 conn(0x7f648c0ffd50 msgr2=0x7f648c102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.982+0000 7f6494600640 1 -- 172.21.15.16:0/848151481 shutdown_connections 2024-04-13T03:51:59.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.982+0000 7f6494600640 1 -- 172.21.15.16:0/848151481 wait complete. 2024-04-13T03:51:59.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.983+0000 7f6494600640 1 Processor -- start 2024-04-13T03:51:59.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.983+0000 7f6494600640 1 -- start start 2024-04-13T03:51:59.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6494600640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648c104470 0x7f648c1a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6494600640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f648c109540 0x7f648c1a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6494600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f648c19a960 0x7f648c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6494600640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f648c114620 con 0x7f648c104470 2024-04-13T03:51:59.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6494600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1144a0 con 0x7f648c19a960 2024-04-13T03:51:59.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6494600640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1147a0 con 0x7f648c109540 2024-04-13T03:51:59.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6491c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648c104470 0x7f648c1a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6491c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648c104470 0x7f648c1a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40736/0 (socket says 172.21.15.16:40736) 2024-04-13T03:51:59.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6491c00640 1 -- 172.21.15.16:0/2387438240 learned_addr learned my addr 172.21.15.16:0/2387438240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:51:59.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6491200640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f648c109540 0x7f648c1a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.984+0000 7f6492600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f648c19a960 0x7f648c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f6492600640 1 -- 172.21.15.16:0/2387438240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f648c109540 msgr2=0x7f648c1a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f6492600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f648c109540 0x7f648c1a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f6492600640 1 -- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648c104470 msgr2=0x7f648c1a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f6492600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648c104470 0x7f648c1a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f6492600640 1 -- 172.21.15.16:0/2387438240 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c19b480 con 0x7f648c19a960 2024-04-13T03:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f6491c00640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648c104470 0x7f648c1a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f6492600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f648c19a960 0x7f648c19ae10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f648400d680 tx=0x7f648400db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f648a000640 1 -- 172.21.15.16:0/2387438240 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484017070 con 0x7f648c19a960 2024-04-13T03:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c19b770 con 0x7f648c19a960 2024-04-13T03:51:59.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f648a000640 1 -- 172.21.15.16:0/2387438240 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6484009820 con 0x7f648c19a960 2024-04-13T03:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f648a000640 1 -- 172.21.15.16:0/2387438240 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484004e00 con 0x7f648c19a960 2024-04-13T03:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.985+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c1a7970 con 0x7f648c19a960 2024-04-13T03:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.987+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c069230 con 0x7f648c19a960 2024-04-13T03:51:59.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.988+0000 7f648a000640 1 -- 172.21.15.16:0/2387438240 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f64840099d0 con 0x7f648c19a960 2024-04-13T03:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.990+0000 7f648a000640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f64740a3d20 0x7f64740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.990+0000 7f648a000640 1 -- 172.21.15.16:0/2387438240 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f64840d05d0 con 0x7f648c19a960 2024-04-13T03:51:59.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.990+0000 7f6491c00640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f64740a3d20 0x7f64740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:51:59.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.991+0000 7f6491c00640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f64740a3d20 0x7f64740a61e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f648c10aa80 tx=0x7f6480009500 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:51:59.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:51:59.993+0000 7f648a000640 1 -- 172.21.15.16:0/2387438240 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f648409c1b0 con 0x7f648c19a960 2024-04-13T03:52:00.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.205+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f648c10c770 con 0x7f648c19a960 2024-04-13T03:52:00.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.206+0000 7f648a000640 1 -- 172.21.15.16:0/2387438240 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v117) ==== 104+0+11 (secure 0 0 0) 0x7f64840a04b0 con 0x7f648c19a960 2024-04-13T03:52:00.205 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:52:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.208+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f64740a3d20 msgr2=0x7f64740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.208+0000 7f6494600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f64740a3d20 0x7f64740a61e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f648c10aa80 tx=0x7f6480009500 comp rx=0 tx=0).stop 2024-04-13T03:52:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f648c19a960 msgr2=0x7f648c19ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f648c19a960 0x7f648c19ae10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f648400d680 tx=0x7f648400db40 comp rx=0 tx=0).stop 2024-04-13T03:52:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 shutdown_connections 2024-04-13T03:52:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f64740a3d20 0x7f64740a61e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f648c19a960 0x7f648c19ae10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f648c109540 0x7f648c1a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 --2- 172.21.15.16:0/2387438240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f648c104470 0x7f648c1a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 >> 172.21.15.16:0/2387438240 conn(0x7f648c0ffd50 msgr2=0x7f648c1020d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 shutdown_connections 2024-04-13T03:52:00.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.209+0000 7f6494600640 1 -- 172.21.15.16:0/2387438240 wait complete. 2024-04-13T03:52:00.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-13T03:52:00.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.478+0000 7fb63f200640 1 -- 172.21.15.16:0/2065945312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb638074360 msgr2=0x7fb638074740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.478+0000 7fb63f200640 1 --2- 172.21.15.16:0/2065945312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb638074360 0x7fb638074740 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb630009be0 tx=0x7fb63002bf80 comp rx=0 tx=0).stop 2024-04-13T03:52:00.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.479+0000 7fb63f200640 1 -- 172.21.15.16:0/2065945312 shutdown_connections 2024-04-13T03:52:00.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.479+0000 7fb63f200640 1 --2- 172.21.15.16:0/2065945312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb638112e70 0x7fb638115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.479+0000 7fb63f200640 1 --2- 172.21.15.16:0/2065945312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb638074c80 0x7fb638112930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.479+0000 7fb63f200640 1 --2- 172.21.15.16:0/2065945312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb638074360 0x7fb638074740 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.479+0000 7fb63f200640 1 -- 172.21.15.16:0/2065945312 >> 172.21.15.16:0/2065945312 conn(0x7fb6380fd990 msgr2=0x7fb6380ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:00.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.479+0000 7fb63f200640 1 -- 172.21.15.16:0/2065945312 shutdown_connections 2024-04-13T03:52:00.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.479+0000 7fb63f200640 1 -- 172.21.15.16:0/2065945312 wait complete. 2024-04-13T03:52:00.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.480+0000 7fb63f200640 1 Processor -- start 2024-04-13T03:52:00.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.480+0000 7fb63f200640 1 -- start start 2024-04-13T03:52:00.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63f200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb638074360 0x7fb63819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:00.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63f200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb638074c80 0x7fb63819f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:00.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63f200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb638112e70 0x7fb6381a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:00.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63f200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb638118890 con 0x7fb638112e70 2024-04-13T03:52:00.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63f200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb638118710 con 0x7fb638074360 2024-04-13T03:52:00.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63f200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fb638118a10 con 0x7fb638074c80 2024-04-13T03:52:00.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb638074c80 0x7fb63819f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:00.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63d400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb638074c80 0x7fb63819f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:33714/0 (socket says 172.21.15.16:33714) 2024-04-13T03:52:00.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63d400640 1 -- 172.21.15.16:0/3387540601 learned_addr learned my addr 172.21.15.16:0/3387540601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:00.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63e800640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb638112e70 0x7fb6381a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:00.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63de00640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb638074360 0x7fb63819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63d400640 1 -- 172.21.15.16:0/3387540601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb638074360 msgr2=0x7fb63819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63d400640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb638074360 0x7fb63819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63d400640 1 -- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb638112e70 msgr2=0x7fb6381a3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63d400640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb638112e70 0x7fb6381a3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63d400640 1 -- 172.21.15.16:0/3387540601 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6381a3da0 con 0x7fb638074c80 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.481+0000 7fb63de00640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb638074360 0x7fb63819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.482+0000 7fb63d400640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb638074c80 0x7fb63819f2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb62800e9c0 tx=0x7fb62800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.482+0000 7fb62ea00640 1 -- 172.21.15.16:0/3387540601 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62800cd80 con 0x7fb638074c80 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.482+0000 7fb62ea00640 1 -- 172.21.15.16:0/3387540601 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6280035f0 con 0x7fb638074c80 2024-04-13T03:52:00.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.482+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6381a4090 con 0x7fb638074c80 2024-04-13T03:52:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.482+0000 7fb62ea00640 1 -- 172.21.15.16:0/3387540601 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb628010670 con 0x7fb638074c80 2024-04-13T03:52:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.482+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6381ab920 con 0x7fb638074c80 2024-04-13T03:52:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.483+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb638068d00 con 0x7fb638074c80 2024-04-13T03:52:00.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.486+0000 7fb62ea00640 1 -- 172.21.15.16:0/3387540601 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb6280037a0 con 0x7fb638074c80 2024-04-13T03:52:00.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.488+0000 7fb62ea00640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb6240a3d20 0x7fb6240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:00.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.488+0000 7fb62ea00640 1 -- 172.21.15.16:0/3387540601 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7fb6280d0f70 con 0x7fb638074c80 2024-04-13T03:52:00.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.488+0000 7fb63de00640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb6240a3d20 0x7fb6240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:00.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.489+0000 7fb63de00640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb6240a3d20 0x7fb6240a61e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb630000f80 tx=0x7fb630009280 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:00.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.489+0000 7fb62ea00640 1 -- 172.21.15.16:0/3387540601 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb62809d1a0 con 0x7fb638074c80 2024-04-13T03:52:00.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:00 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1387282549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:52:00.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:00 smithi119 ceph-mon[35835]: pgmap v542: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 14 op/s 2024-04-13T03:52:00.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:00 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2387438240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:52:00.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:00 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1387282549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:52:00.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:00 smithi016 ceph-mon[31772]: pgmap v542: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 14 op/s 2024-04-13T03:52:00.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:00 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2387438240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:52:00.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:00 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1387282549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-13T03:52:00.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:00 smithi188 ceph-mon[36398]: pgmap v542: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 14 op/s 2024-04-13T03:52:00.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:00 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2387438240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-13T03:52:00.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.699+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb638105360 con 0x7fb638074c80 2024-04-13T03:52:00.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.699+0000 7fb62ea00640 1 -- 172.21.15.16:0/3387540601 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v117) ==== 100+0+11 (secure 0 0 0) 0x7fb62809caa0 con 0x7fb638074c80 2024-04-13T03:52:00.699 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-13T03:52:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb6240a3d20 msgr2=0x7fb6240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb6240a3d20 0x7fb6240a61e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb630000f80 tx=0x7fb630009280 comp rx=0 tx=0).stop 2024-04-13T03:52:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb638074c80 msgr2=0x7fb63819f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb638074c80 0x7fb63819f2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb62800e9c0 tx=0x7fb62800edc0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 shutdown_connections 2024-04-13T03:52:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fb6240a3d20 0x7fb6240a61e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb638112e70 0x7fb6381a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb638074c80 0x7fb63819f2f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 --2- 172.21.15.16:0/3387540601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb638074360 0x7fb63819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 >> 172.21.15.16:0/3387540601 conn(0x7fb6380fd990 msgr2=0x7fb6380fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.702+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 shutdown_connections 2024-04-13T03:52:00.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.703+0000 7fb63f200640 1 -- 172.21.15.16:0/3387540601 wait complete. 2024-04-13T03:52:00.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-13T03:52:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.953+0000 7f97cdc00640 1 -- 172.21.15.16:0/2529427861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97c8103090 msgr2=0x7f97c8103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.953+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2529427861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97c8103090 0x7f97c8103470 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009860 tx=0x7f97b802bc10 comp rx=0 tx=0).stop 2024-04-13T03:52:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.954+0000 7f97cdc00640 1 -- 172.21.15.16:0/2529427861 shutdown_connections 2024-04-13T03:52:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.954+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2529427861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97c810c4b0 0x7f97c810e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.954+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2529427861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c81039b0 0x7f97c810be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.954+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2529427861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97c8103090 0x7f97c8103470 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.954+0000 7f97cdc00640 1 -- 172.21.15.16:0/2529427861 >> 172.21.15.16:0/2529427861 conn(0x7f97c80fc8d0 msgr2=0x7f97c80fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:00.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.954+0000 7f97cdc00640 1 -- 172.21.15.16:0/2529427861 shutdown_connections 2024-04-13T03:52:00.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.954+0000 7f97cdc00640 1 -- 172.21.15.16:0/2529427861 wait complete. 2024-04-13T03:52:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97cdc00640 1 Processor -- start 2024-04-13T03:52:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97cdc00640 1 -- start start 2024-04-13T03:52:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97cdc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97c8103090 0x7f97c81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97cdc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c81039b0 0x7f97c81a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:00.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97cdc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97c810c4b0 0x7f97c819a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97cdc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8112290 con 0x7f97c81039b0 2024-04-13T03:52:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97cdc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8112110 con 0x7f97c810c4b0 2024-04-13T03:52:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97cdc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8112410 con 0x7f97c8103090 2024-04-13T03:52:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97bfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c81039b0 0x7f97c81a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:00.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.955+0000 7f97cd200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97c810c4b0 0x7f97c819a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97bfe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c81039b0 0x7f97c81a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40762/0 (socket says 172.21.15.16:40762) 2024-04-13T03:52:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97bfe00640 1 -- 172.21.15.16:0/2891300670 learned_addr learned my addr 172.21.15.16:0/2891300670 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97c7e00640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97c8103090 0x7f97c81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97cd200640 1 -- 172.21.15.16:0/2891300670 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97c8103090 msgr2=0x7f97c81a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97cd200640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97c8103090 0x7f97c81a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97cd200640 1 -- 172.21.15.16:0/2891300670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c81039b0 msgr2=0x7f97c81a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97cd200640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c81039b0 0x7f97c81a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97cd200640 1 -- 172.21.15.16:0/2891300670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c819b030 con 0x7f97c810c4b0 2024-04-13T03:52:00.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97cd200640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97c810c4b0 0x7f97c819a8b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009fd0 tx=0x7f97c000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.956+0000 7f97c5600640 1 -- 172.21.15.16:0/2891300670 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c000ce20 con 0x7f97c810c4b0 2024-04-13T03:52:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.957+0000 7f97c5600640 1 -- 172.21.15.16:0/2891300670 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c0005020 con 0x7f97c810c4b0 2024-04-13T03:52:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.957+0000 7f97c5600640 1 -- 172.21.15.16:0/2891300670 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0010720 con 0x7f97c810c4b0 2024-04-13T03:52:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.957+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c819b320 con 0x7f97c810c4b0 2024-04-13T03:52:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.957+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c81a74d0 con 0x7f97c810c4b0 2024-04-13T03:52:00.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.958+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c806aea0 con 0x7f97c810c4b0 2024-04-13T03:52:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.960+0000 7f97c5600640 1 -- 172.21.15.16:0/2891300670 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f97c0005540 con 0x7f97c810c4b0 2024-04-13T03:52:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.961+0000 7f97c5600640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f97ac0a3da0 0x7f97ac0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.962+0000 7f97c5600640 1 -- 172.21.15.16:0/2891300670 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f97c00d1220 con 0x7f97c810c4b0 2024-04-13T03:52:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.962+0000 7f97c7e00640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f97ac0a3da0 0x7f97ac0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.962+0000 7f97c7e00640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f97ac0a3da0 0x7f97ac0a6260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009830 tx=0x7f97b80056d0 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:00.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:00.964+0000 7f97c5600640 1 -- 172.21.15.16:0/2891300670 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97c009d4e0 con 0x7f97c810c4b0 2024-04-13T03:52:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.172+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f97c81a7700 con 0x7f97c810c4b0 2024-04-13T03:52:01.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.173+0000 7f97c5600640 1 -- 172.21.15.16:0/2891300670 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v117) ==== 103+0+11 (secure 0 0 0) 0x7f97c009cde0 con 0x7f97c810c4b0 2024-04-13T03:52:01.172 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-13T03:52:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f97ac0a3da0 msgr2=0x7f97ac0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f97ac0a3da0 0x7f97ac0a6260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009830 tx=0x7f97b80056d0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97c810c4b0 msgr2=0x7f97c819a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97c810c4b0 0x7f97c819a8b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f97c0009fd0 tx=0x7f97c000ef10 comp rx=0 tx=0).stop 2024-04-13T03:52:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 shutdown_connections 2024-04-13T03:52:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f97ac0a3da0 0x7f97ac0a6260 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97c810c4b0 0x7f97c819a8b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f97c81039b0 0x7f97c81a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 --2- 172.21.15.16:0/2891300670 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97c8103090 0x7f97c81a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 >> 172.21.15.16:0/2891300670 conn(0x7f97c80fc8d0 msgr2=0x7f97c8101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 shutdown_connections 2024-04-13T03:52:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.176+0000 7f97cdc00640 1 -- 172.21.15.16:0/2891300670 wait complete. 2024-04-13T03:52:01.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-13T03:52:01.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.422+0000 7f7a0fa00640 1 -- 172.21.15.16:0/3223010777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a081072f0 msgr2=0x7f7a0810eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:01.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.422+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/3223010777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a081072f0 0x7f7a0810eea0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009be0 tx=0x7f7a0002bef0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.422+0000 7f7a0fa00640 1 -- 172.21.15.16:0/3223010777 shutdown_connections 2024-04-13T03:52:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.422+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/3223010777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a081072f0 0x7f7a0810eea0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.422+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/3223010777 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a0806b1e0 0x7f7a08106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.422+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/3223010777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0806a830 0x7f7a0806ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.422+0000 7f7a0fa00640 1 -- 172.21.15.16:0/3223010777 >> 172.21.15.16:0/3223010777 conn(0x7f7a08076cc0 msgr2=0x7f7a080770d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.423+0000 7f7a0fa00640 1 -- 172.21.15.16:0/3223010777 shutdown_connections 2024-04-13T03:52:01.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.423+0000 7f7a0fa00640 1 -- 172.21.15.16:0/3223010777 wait complete. 2024-04-13T03:52:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0fa00640 1 Processor -- start 2024-04-13T03:52:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0fa00640 1 -- start start 2024-04-13T03:52:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0fa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0806a830 0x7f7a0819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0fa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a081072f0 0x7f7a0819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0fa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a08198630 0x7f7a08198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0fa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7a081123f0 con 0x7f7a0806a830 2024-04-13T03:52:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0fa00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7a08112270 con 0x7f7a081072f0 2024-04-13T03:52:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0fa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f7a08112570 con 0x7f7a08198630 2024-04-13T03:52:01.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0806a830 0x7f7a0819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:01.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0d000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0806a830 0x7f7a0819e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40782/0 (socket says 172.21.15.16:40782) 2024-04-13T03:52:01.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.424+0000 7f7a0d000640 1 -- 172.21.15.16:0/652646343 learned_addr learned my addr 172.21.15.16:0/652646343 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:01.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.425+0000 7f7a07e00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a081072f0 0x7f7a0819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:01.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.425+0000 7f7a0da00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a08198630 0x7f7a08198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:01.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.425+0000 7f7a0d000640 1 -- 172.21.15.16:0/652646343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a08198630 msgr2=0x7f7a08198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.425+0000 7f7a0d000640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a08198630 0x7f7a08198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.425+0000 7f7a0d000640 1 -- 172.21.15.16:0/652646343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a081072f0 msgr2=0x7f7a0819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.425+0000 7f7a0d000640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a081072f0 0x7f7a0819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.425+0000 7f7a0d000640 1 -- 172.21.15.16:0/652646343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a08199230 con 0x7f7a0806a830 2024-04-13T03:52:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.425+0000 7f7a0d000640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0806a830 0x7f7a0819e570 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f79f800e9c0 tx=0x7f79f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.426+0000 7f7a05600640 1 -- 172.21.15.16:0/652646343 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f800ccc0 con 0x7f7a0806a830 2024-04-13T03:52:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.426+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a081a5310 con 0x7f7a0806a830 2024-04-13T03:52:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.426+0000 7f7a05600640 1 -- 172.21.15.16:0/652646343 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79f8005500 con 0x7f7a0806a830 2024-04-13T03:52:01.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.426+0000 7f7a05600640 1 -- 172.21.15.16:0/652646343 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f8010430 con 0x7f7a0806a830 2024-04-13T03:52:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.426+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a081a5770 con 0x7f7a0806a830 2024-04-13T03:52:01.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.427+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a081036f0 con 0x7f7a0806a830 2024-04-13T03:52:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.429+0000 7f7a05600640 1 -- 172.21.15.16:0/652646343 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f79f80046c0 con 0x7f7a0806a830 2024-04-13T03:52:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.430+0000 7f7a05600640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f79ec0a3d20 0x7f79ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.431+0000 7f7a05600640 1 -- 172.21.15.16:0/652646343 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f79f80d0f50 con 0x7f7a0806a830 2024-04-13T03:52:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.431+0000 7f7a07e00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f79ec0a3d20 0x7f79ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:01.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.431+0000 7f7a07e00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f79ec0a3d20 0x7f79ec0a61e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a08199e80 tx=0x7f79f0005f90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:01.434 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.435+0000 7f7a05600640 1 -- 172.21.15.16:0/652646343 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79f809d180 con 0x7f7a0806a830 2024-04-13T03:52:01.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:01 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3387540601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:52:01.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:01 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2891300670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:52:01.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:01 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3387540601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:52:01.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:01 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2891300670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:52:01.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.660+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7a0806b270 con 0x7f7a0806a830 2024-04-13T03:52:01.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.661+0000 7f7a05600640 1 -- 172.21.15.16:0/652646343 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v117) ==== 81+0+940 (secure 0 0 0) 0x7f79f809ca80 con 0x7f7a0806a830 2024-04-13T03:52:01.661 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2756805961 2024-04-14T03:34:45.365818+0000 2024-04-13T03:52:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2348781121 2024-04-14T03:34:45.365818+0000 2024-04-13T03:52:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1183476525 2024-04-14T03:34:45.365818+0000 2024-04-13T03:52:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/1721562410 2024-04-14T03:34:45.365818+0000 2024-04-13T03:52:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2242726589 2024-04-14T03:34:45.365818+0000 2024-04-13T03:52:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/1721562410 2024-04-14T03:34:45.365818+0000 2024-04-13T03:52:01.662 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/164620940 2024-04-14T03:23:39.714975+0000 2024-04-13T03:52:01.663 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/83197818 2024-04-14T03:23:04.480630+0000 2024-04-13T03:52:01.663 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/20480070 2024-04-14T03:23:39.714975+0000 2024-04-13T03:52:01.663 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3152772162 2024-04-14T03:23:04.480630+0000 2024-04-13T03:52:01.663 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/2135228404 2024-04-14T03:23:04.480630+0000 2024-04-13T03:52:01.663 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/2468290259 2024-04-14T03:23:39.714975+0000 2024-04-13T03:52:01.663 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6801/164620940 2024-04-14T03:23:39.714975+0000 2024-04-13T03:52:01.663 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/1664031642 2024-04-14T03:23:04.480630+0000 2024-04-13T03:52:01.664 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:0/3312707257 2024-04-14T03:23:39.714975+0000 2024-04-13T03:52:01.664 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6800/2135228404 2024-04-14T03:23:04.480630+0000 2024-04-13T03:52:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:listed 16 entries 2024-04-13T03:52:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.663+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f79ec0a3d20 msgr2=0x7f79ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.663+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f79ec0a3d20 0x7f79ec0a61e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a08199e80 tx=0x7f79f0005f90 comp rx=0 tx=0).stop 2024-04-13T03:52:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.663+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0806a830 msgr2=0x7f7a0819e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.663+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0806a830 0x7f7a0819e570 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f79f800e9c0 tx=0x7f79f800ee80 comp rx=0 tx=0).stop 2024-04-13T03:52:01.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.664+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 shutdown_connections 2024-04-13T03:52:01.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.664+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f79ec0a3d20 0x7f79ec0a61e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.664+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7a08198630 0x7f7a08198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.664+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a081072f0 0x7f7a0819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.664+0000 7f7a0fa00640 1 --2- 172.21.15.16:0/652646343 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a0806a830 0x7f7a0819e570 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:01.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.664+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 >> 172.21.15.16:0/652646343 conn(0x7f7a08076cc0 msgr2=0x7f7a080ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:01.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.664+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 shutdown_connections 2024-04-13T03:52:01.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:01.664+0000 7f7a0fa00640 1 -- 172.21.15.16:0/652646343 wait complete. 2024-04-13T03:52:01.684 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-13T03:52:01.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:52:01.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-13T03:52:01.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:01 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3387540601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-13T03:52:01.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:01 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2891300670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-13T03:52:01.968 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T03:52:01.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:52:01.968 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-13T03:52:02.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:52:02.052 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-13T03:52:02.137 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-13T03:52:02.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:52:02.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-13T03:52:02.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-13T03:52:02.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-13T03:52:02.345 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-13T03:52:02.356 INFO:tasks.cephadm:Teardown begin 2024-04-13T03:52:02.356 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-13T03:52:02.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:52:02.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.539+0000 7f6e8cc00640 1 -- 172.21.15.16:0/581887250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e88109300 msgr2=0x7f6e88110ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:02.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.539+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/581887250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e88109300 0x7f6e88110ed0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009be0 tx=0x7f6e8002bef0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.540+0000 7f6e8cc00640 1 -- 172.21.15.16:0/581887250 shutdown_connections 2024-04-13T03:52:02.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.540+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/581887250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e88109300 0x7f6e88110ed0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.540+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/581887250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e88104be0 0x7f6e88108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.540+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/581887250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e88104230 0x7f6e88104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.540+0000 7f6e8cc00640 1 -- 172.21.15.16:0/581887250 >> 172.21.15.16:0/581887250 conn(0x7f6e880ffa90 msgr2=0x7f6e88101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:02.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.540+0000 7f6e8cc00640 1 -- 172.21.15.16:0/581887250 shutdown_connections 2024-04-13T03:52:02.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.540+0000 7f6e8cc00640 1 -- 172.21.15.16:0/581887250 wait complete. 2024-04-13T03:52:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e8cc00640 1 Processor -- start 2024-04-13T03:52:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e8cc00640 1 -- start start 2024-04-13T03:52:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e8cc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e88104230 0x7f6e881a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e8cc00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e88109300 0x7f6e881a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e8cc00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e8819a680 0x7f6e8819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e8cc00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6e881143e0 con 0x7f6e8819a680 2024-04-13T03:52:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e8cc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6e88114260 con 0x7f6e88104230 2024-04-13T03:52:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e8cc00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f6e88114560 con 0x7f6e88109300 2024-04-13T03:52:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e86a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e88109300 0x7f6e881a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e87e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e8819a680 0x7f6e8819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e86a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e88109300 0x7f6e881a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:33768/0 (socket says 172.21.15.16:33768) 2024-04-13T03:52:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e86a00640 1 -- 172.21.15.16:0/3090137366 learned_addr learned my addr 172.21.15.16:0/3090137366 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.541+0000 7f6e87400640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e88104230 0x7f6e881a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e87e00640 1 -- 172.21.15.16:0/3090137366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e88109300 msgr2=0x7f6e881a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e87e00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e88109300 0x7f6e881a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e87e00640 1 -- 172.21.15.16:0/3090137366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e88104230 msgr2=0x7f6e881a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e87e00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e88104230 0x7f6e881a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e87e00640 1 -- 172.21.15.16:0/3090137366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8819b280 con 0x7f6e8819a680 2024-04-13T03:52:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e87e00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e8819a680 0x7f6e8819ab00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6e8002c430 tx=0x7f6e800336b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e7fe00640 1 -- 172.21.15.16:0/3090137366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80043020 con 0x7f6e8819a680 2024-04-13T03:52:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e7fe00640 1 -- 172.21.15.16:0/3090137366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e80003e50 con 0x7f6e8819a680 2024-04-13T03:52:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e7fe00640 1 -- 172.21.15.16:0/3090137366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8003e450 con 0x7f6e8819a680 2024-04-13T03:52:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.542+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e8819b510 con 0x7f6e8819a680 2024-04-13T03:52:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.543+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e881a7660 con 0x7f6e8819a680 2024-04-13T03:52:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.544+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e88105700 con 0x7f6e8819a680 2024-04-13T03:52:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.546+0000 7f6e7fe00640 1 -- 172.21.15.16:0/3090137366 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6e80046020 con 0x7f6e8819a680 2024-04-13T03:52:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.547+0000 7f6e7fe00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f6e6c0a3d20 0x7f6e6c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:02.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.547+0000 7f6e7fe00640 1 -- 172.21.15.16:0/3090137366 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f6e800f3840 con 0x7f6e8819a680 2024-04-13T03:52:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.547+0000 7f6e87400640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f6e6c0a3d20 0x7f6e6c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.548+0000 7f6e87400640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f6e6c0a3d20 0x7f6e6c0a61e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009bd0 tx=0x7f6e78005f90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:02.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.550+0000 7f6e7fe00640 1 -- 172.21.15.16:0/3090137366 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e8004d020 con 0x7f6e8819a680 2024-04-13T03:52:02.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:02 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/652646343' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-13T03:52:02.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:02 smithi016 ceph-mon[31772]: pgmap v543: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 148 KiB/s wr, 22 op/s 2024-04-13T03:52:02.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:02 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/652646343' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-13T03:52:02.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:02 smithi188 ceph-mon[36398]: pgmap v543: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 148 KiB/s wr, 22 op/s 2024-04-13T03:52:02.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.799+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e88069150 con 0x7f6e8819a680 2024-04-13T03:52:02.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.800+0000 7f6e7fe00640 1 -- 172.21.15.16:0/3090137366 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+11934 (secure 0 0 0) 0x7f6e800bfd00 con 0x7f6e8819a680 2024-04-13T03:52:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:52:02.801 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":73,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:51:49.728724+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24530,"mds_2":24523,"mds_3":24505,"mds_4":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24523,24530,24587,24500,24505]},"id":1}]} 2024-04-13T03:52:02.802 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2024-04-13T03:52:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f6e6c0a3d20 msgr2=0x7f6e6c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f6e6c0a3d20 0x7f6e6c0a61e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009bd0 tx=0x7f6e78005f90 comp rx=0 tx=0).stop 2024-04-13T03:52:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e8819a680 msgr2=0x7f6e8819ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e8819a680 0x7f6e8819ab00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6e8002c430 tx=0x7f6e800336b0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 shutdown_connections 2024-04-13T03:52:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f6e6c0a3d20 0x7f6e6c0a61e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e8819a680 0x7f6e8819ab00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6e88109300 0x7f6e881a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 --2- 172.21.15.16:0/3090137366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e88104230 0x7f6e881a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:02.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 >> 172.21.15.16:0/3090137366 conn(0x7f6e880ffa90 msgr2=0x7f6e88100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:02.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.803+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 shutdown_connections 2024-04-13T03:52:02.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:02.804+0000 7f6e8cc00640 1 -- 172.21.15.16:0/3090137366 wait complete. 2024-04-13T03:52:02.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:02 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/652646343' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-13T03:52:02.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:02 smithi119 ceph-mon[35835]: pgmap v543: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 148 KiB/s wr, 22 op/s 2024-04-13T03:52:02.823 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-13T03:52:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.040+0000 7f1d00c00640 1 -- 172.21.15.16:0/1269820044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cfc107160 msgr2=0x7f1cfc10ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.040+0000 7f1d00c00640 1 --2- 172.21.15.16:0/1269820044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cfc107160 0x7f1cfc10ed00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009be0 tx=0x7f1cf402bef0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.040+0000 7f1d00c00640 1 -- 172.21.15.16:0/1269820044 shutdown_connections 2024-04-13T03:52:03.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.040+0000 7f1d00c00640 1 --2- 172.21.15.16:0/1269820044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cfc107160 0x7f1cfc10ed00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.040+0000 7f1d00c00640 1 --2- 172.21.15.16:0/1269820044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cfc069d70 0x7f1cfc106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.040+0000 7f1d00c00640 1 --2- 172.21.15.16:0/1269820044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc0693c0 0x7f1cfc0697a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.040+0000 7f1d00c00640 1 -- 172.21.15.16:0/1269820044 >> 172.21.15.16:0/1269820044 conn(0x7f1cfc076800 msgr2=0x7f1cfc076c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.040+0000 7f1d00c00640 1 -- 172.21.15.16:0/1269820044 shutdown_connections 2024-04-13T03:52:03.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.040+0000 7f1d00c00640 1 -- 172.21.15.16:0/1269820044 wait complete. 2024-04-13T03:52:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.041+0000 7f1d00c00640 1 Processor -- start 2024-04-13T03:52:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1d00c00640 1 -- start start 2024-04-13T03:52:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1d00c00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc0693c0 0x7f1cfc1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1d00c00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cfc107160 0x7f1cfc1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1d00c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cfc19a6d0 0x7f1cfc19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1d00c00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc1121f0 con 0x7f1cfc107160 2024-04-13T03:52:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1d00c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc112070 con 0x7f1cfc19a6d0 2024-04-13T03:52:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1d00c00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc112370 con 0x7f1cfc0693c0 2024-04-13T03:52:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1cfb400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc0693c0 0x7f1cfc1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1cf2a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cfc107160 0x7f1cfc1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:03.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1cfb400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc0693c0 0x7f1cfc1a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:33780/0 (socket says 172.21.15.16:33780) 2024-04-13T03:52:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1cfb400640 1 -- 172.21.15.16:0/2024431419 learned_addr learned my addr 172.21.15.16:0/2024431419 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.042+0000 7f1cfbe00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cfc19a6d0 0x7f1cfc19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1cfb400640 1 -- 172.21.15.16:0/2024431419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cfc19a6d0 msgr2=0x7f1cfc19ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1cfb400640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cfc19a6d0 0x7f1cfc19ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1cfb400640 1 -- 172.21.15.16:0/2024431419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cfc107160 msgr2=0x7f1cfc1a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1cfb400640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cfc107160 0x7f1cfc1a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1cfb400640 1 -- 172.21.15.16:0/2024431419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cfc19b2d0 con 0x7f1cfc0693c0 2024-04-13T03:52:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1cfb400640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc0693c0 0x7f1cfc1a0610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00e9f0 tx=0x7f1cec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1cf8c00640 1 -- 172.21.15.16:0/2024431419 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec00ccc0 con 0x7f1cfc0693c0 2024-04-13T03:52:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cfc1a73b0 con 0x7f1cfc0693c0 2024-04-13T03:52:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1cf8c00640 1 -- 172.21.15.16:0/2024431419 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cec005500 con 0x7f1cfc0693c0 2024-04-13T03:52:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.043+0000 7f1cf8c00640 1 -- 172.21.15.16:0/2024431419 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec010690 con 0x7f1cfc0693c0 2024-04-13T03:52:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.044+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cfc1a7760 con 0x7f1cfc0693c0 2024-04-13T03:52:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.045+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cfc103510 con 0x7f1cfc0693c0 2024-04-13T03:52:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.047+0000 7f1cf8c00640 1 -- 172.21.15.16:0/2024431419 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1cec0046c0 con 0x7f1cfc0693c0 2024-04-13T03:52:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.048+0000 7f1cf8c00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f1ce00a3d20 0x7f1ce00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.048+0000 7f1cf8c00640 1 -- 172.21.15.16:0/2024431419 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f1cec0d10e0 con 0x7f1cfc0693c0 2024-04-13T03:52:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.049+0000 7f1cf2a00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f1ce00a3d20 0x7f1ce00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:03.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.049+0000 7f1cf2a00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f1ce00a3d20 0x7f1ce00a61e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc19bf20 tx=0x7f1ce4005f90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:03.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.050+0000 7f1cf8c00640 1 -- 172.21.15.16:0/2024431419 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cec053ff0 con 0x7f1cfc0693c0 2024-04-13T03:52:03.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.288+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cfc068d00 con 0x7f1cfc0693c0 2024-04-13T03:52:03.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.290+0000 7f1cf8c00640 1 -- 172.21.15.16:0/2024431419 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+11934 (secure 0 0 0) 0x7f1cec09d310 con 0x7f1cfc0693c0 2024-04-13T03:52:03.292 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:52:03.292 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6839/3959195994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6838","nonce":3959195994},{"type":"v1","addr":"172.21.15.16:6839","nonce":3959195994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6841/701411811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6840","nonce":701411811},{"type":"v1","addr":"172.21.15.16:6841","nonce":701411811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":73,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-13T03:31:37.109492+0000","modified":"2024-04-13T03:51:49.728724+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24587,"mds_1":24530,"mds_2":24523,"mds_3":24505,"mds_4":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24433":{"gid":24433,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6833/3805092939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":3805092939},{"type":"v1","addr":"172.21.15.188:6833","nonce":3805092939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/3221738952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3221738952},{"type":"v1","addr":"172.21.15.119:6833","nonce":3221738952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6835/1177424525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6834","nonce":1177424525},{"type":"v1","addr":"172.21.15.16:6835","nonce":1177424525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.16:6837/2782624805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6836","nonce":2782624805},{"type":"v1","addr":"172.21.15.16:6837","nonce":2782624805}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6835/566226134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":566226134},{"type":"v1","addr":"172.21.15.188:6835","nonce":566226134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/4186042513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4186042513},{"type":"v1","addr":"172.21.15.119:6835","nonce":4186042513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.188:6837/284865646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":284865646},{"type":"v1","addr":"172.21.15.188:6837","nonce":284865646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.188:6839/57242391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":57242391},{"type":"v1","addr":"172.21.15.188:6839","nonce":57242391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.119:6837/477256695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":477256695},{"type":"v1","addr":"172.21.15.119:6837","nonce":477256695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/3722020849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":3722020849},{"type":"v1","addr":"172.21.15.119:6839","nonce":3722020849}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24523,24530,24587,24500,24505]},"id":1}]} 2024-04-13T03:52:03.292 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2024-04-13T03:52:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f1ce00a3d20 msgr2=0x7f1ce00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f1ce00a3d20 0x7f1ce00a61e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc19bf20 tx=0x7f1ce4005f90 comp rx=0 tx=0).stop 2024-04-13T03:52:03.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc0693c0 msgr2=0x7f1cfc1a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc0693c0 0x7f1cfc1a0610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00e9f0 tx=0x7f1cec00eeb0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 shutdown_connections 2024-04-13T03:52:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f1ce00a3d20 0x7f1ce00a61e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cfc19a6d0 0x7f1cfc19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cfc107160 0x7f1cfc1a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 --2- 172.21.15.16:0/2024431419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cfc0693c0 0x7f1cfc1a0610 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.292+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 >> 172.21.15.16:0/2024431419 conn(0x7f1cfc076800 msgr2=0x7f1cfc100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.293+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 shutdown_connections 2024-04-13T03:52:03.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.293+0000 7f1d00c00640 1 -- 172.21.15.16:0/2024431419 wait complete. 2024-04-13T03:52:03.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-13T03:52:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.533+0000 7f3f33200640 1 -- 172.21.15.16:0/1912079179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34105cf0 msgr2=0x7f3f341060d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.533+0000 7f3f33200640 1 --2- 172.21.15.16:0/1912079179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34105cf0 0x7f3f341060d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009ca0 tx=0x7f3f2402c180 comp rx=0 tx=0).stop 2024-04-13T03:52:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.533+0000 7f3f33200640 1 -- 172.21.15.16:0/1912079179 shutdown_connections 2024-04-13T03:52:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.533+0000 7f3f33200640 1 --2- 172.21.15.16:0/1912079179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f34101970 0x7f3f3410db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.533+0000 7f3f33200640 1 --2- 172.21.15.16:0/1912079179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f34100fb0 0x7f3f34101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.533+0000 7f3f33200640 1 --2- 172.21.15.16:0/1912079179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34105cf0 0x7f3f341060d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.533+0000 7f3f33200640 1 -- 172.21.15.16:0/1912079179 >> 172.21.15.16:0/1912079179 conn(0x7f3f34076800 msgr2=0x7f3f34076c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.533+0000 7f3f33200640 1 -- 172.21.15.16:0/1912079179 shutdown_connections 2024-04-13T03:52:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.533+0000 7f3f33200640 1 -- 172.21.15.16:0/1912079179 wait complete. 2024-04-13T03:52:03.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.534+0000 7f3f33200640 1 Processor -- start 2024-04-13T03:52:03.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.534+0000 7f3f33200640 1 -- start start 2024-04-13T03:52:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f33200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34100fb0 0x7f3f341a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f33200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f34101970 0x7f3f341a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f33200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f34105cf0 0x7f3f3419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f33200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f341110a0 con 0x7f3f34100fb0 2024-04-13T03:52:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f33200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3f34110f20 con 0x7f3f34101970 2024-04-13T03:52:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f33200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3f34111220 con 0x7f3f34105cf0 2024-04-13T03:52:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f31e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34100fb0 0x7f3f341a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:03.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f31e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34100fb0 0x7f3f341a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40826/0 (socket says 172.21.15.16:40826) 2024-04-13T03:52:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f31e00640 1 -- 172.21.15.16:0/1970638480 learned_addr learned my addr 172.21.15.16:0/1970638480 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f32800640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f34105cf0 0x7f3f3419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.535+0000 7f3f31400640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f34101970 0x7f3f341a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f31e00640 1 -- 172.21.15.16:0/1970638480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f34105cf0 msgr2=0x7f3f3419a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f31e00640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f34105cf0 0x7f3f3419a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f31e00640 1 -- 172.21.15.16:0/1970638480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f34101970 msgr2=0x7f3f341a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f31e00640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f34101970 0x7f3f341a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f31e00640 1 -- 172.21.15.16:0/1970638480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3419b040 con 0x7f3f34100fb0 2024-04-13T03:52:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f31e00640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34100fb0 0x7f3f341a0670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3f24000f80 tx=0x7f3f24034650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f2aa00640 1 -- 172.21.15.16:0/1970638480 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24039050 con 0x7f3f34100fb0 2024-04-13T03:52:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f2aa00640 1 -- 172.21.15.16:0/1970638480 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f24045070 con 0x7f3f34100fb0 2024-04-13T03:52:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f2aa00640 1 -- 172.21.15.16:0/1970638480 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24004380 con 0x7f3f34100fb0 2024-04-13T03:52:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3419b2d0 con 0x7f3f34100fb0 2024-04-13T03:52:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.536+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f341a75f0 con 0x7f3f34100fb0 2024-04-13T03:52:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.537+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f34102420 con 0x7f3f34100fb0 2024-04-13T03:52:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.539+0000 7f3f2aa00640 1 -- 172.21.15.16:0/1970638480 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3f24004520 con 0x7f3f34100fb0 2024-04-13T03:52:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.542+0000 7f3f2aa00640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3f180a3da0 0x7f3f180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.542+0000 7f3f2aa00640 1 -- 172.21.15.16:0/1970638480 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f3f240f30c0 con 0x7f3f34100fb0 2024-04-13T03:52:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.542+0000 7f3f31400640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3f180a3da0 0x7f3f180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.542+0000 7f3f31400640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3f180a3da0 0x7f3f180a6260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3f3419bf80 tx=0x7f3f1c005f90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:03.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.547+0000 7f3f2aa00640 1 -- 172.21.15.16:0/1970638480 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f240333a0 con 0x7f3f34100fb0 2024-04-13T03:52:03.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:03 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3090137366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:52:03.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:03 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2024431419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:52:03.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:03 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3090137366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:52:03.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:03 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2024431419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:52:03.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.754+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f341a77c0 con 0x7f3f34100fb0 2024-04-13T03:52:03.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.758+0000 7f3f2aa00640 1 -- 172.21.15.16:0/1970638480 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) ==== 74+0+25440 (secure 0 0 0) 0x7f3f240bf400 con 0x7f3f34100fb0 2024-04-13T03:52:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-13T03:52:03.760 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"fsid":"e6bf0314-f944-11ee-bc8d-c7b262605968","created":"2024-04-13T03:22:22.658381+0000","modified":"2024-04-13T03:51:08.244014+0000","last_up_change":"2024-04-13T03:30:38.654877+0000","last_in_change":"2024-04-13T03:30:19.034243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-13T03:26:21.881220+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-13T03:31:34.802688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-13T03:31:36.036786+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-13T03:31:39.824413+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":116,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"69f3ffdc-4c06-47c2-92c0-1b7aab496918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6803","nonce":4119728361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6805","nonce":4119728361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6809","nonce":4119728361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4119728361},{"type":"v1","addr":"172.21.15.16:6807","nonce":4119728361}]},"public_addr":"172.21.15.16:6803/4119728361","cluster_addr":"172.21.15.16:6805/4119728361","heartbeat_back_addr":"172.21.15.16:6809/4119728361","heartbeat_front_addr":"172.21.15.16:6807/4119728361","state":["exists","up"]},{"osd":1,"uuid":"f603a5d2-28ba-439b-8855-6ff82db896aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6801","nonce":1083610996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6803","nonce":1083610996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6807","nonce":1083610996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1083610996},{"type":"v1","addr":"172.21.15.119:6805","nonce":1083610996}]},"public_addr":"172.21.15.119:6801/1083610996","cluster_addr":"172.21.15.119:6803/1083610996","heartbeat_back_addr":"172.21.15.119:6807/1083610996","heartbeat_front_addr":"172.21.15.119:6805/1083610996","state":["exists","up"]},{"osd":2,"uuid":"3c288439-37d1-43e9-8e52-9a08bdbcfff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6801","nonce":1923423179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6803","nonce":1923423179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6807","nonce":1923423179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1923423179},{"type":"v1","addr":"172.21.15.188:6805","nonce":1923423179}]},"public_addr":"172.21.15.188:6801/1923423179","cluster_addr":"172.21.15.188:6803/1923423179","heartbeat_back_addr":"172.21.15.188:6807/1923423179","heartbeat_front_addr":"172.21.15.188:6805/1923423179","state":["exists","up"]},{"osd":3,"uuid":"a289feb8-b06b-4faf-b5f1-bf88a3cc8b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6811","nonce":3993319760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6813","nonce":3993319760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6817","nonce":3993319760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":3993319760},{"type":"v1","addr":"172.21.15.16:6815","nonce":3993319760}]},"public_addr":"172.21.15.16:6811/3993319760","cluster_addr":"172.21.15.16:6813/3993319760","heartbeat_back_addr":"172.21.15.16:6817/3993319760","heartbeat_front_addr":"172.21.15.16:6815/3993319760","state":["exists","up"]},{"osd":4,"uuid":"d1925505-90c0-4946-b886-25813ecd4c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6809","nonce":1038380555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6811","nonce":1038380555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6815","nonce":1038380555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1038380555},{"type":"v1","addr":"172.21.15.119:6813","nonce":1038380555}]},"public_addr":"172.21.15.119:6809/1038380555","cluster_addr":"172.21.15.119:6811/1038380555","heartbeat_back_addr":"172.21.15.119:6815/1038380555","heartbeat_front_addr":"172.21.15.119:6813/1038380555","state":["exists","up"]},{"osd":5,"uuid":"c8aeb2ab-718c-4ff2-9bda-2b28cbf90ea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6809","nonce":2068292885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6811","nonce":2068292885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6815","nonce":2068292885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":2068292885},{"type":"v1","addr":"172.21.15.188:6813","nonce":2068292885}]},"public_addr":"172.21.15.188:6809/2068292885","cluster_addr":"172.21.15.188:6811/2068292885","heartbeat_back_addr":"172.21.15.188:6815/2068292885","heartbeat_front_addr":"172.21.15.188:6813/2068292885","state":["exists","up"]},{"osd":6,"uuid":"facc0570-ab47-4224-9ac9-290d12b6a5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6819","nonce":1633490210}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6821","nonce":1633490210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6825","nonce":1633490210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1633490210},{"type":"v1","addr":"172.21.15.16:6823","nonce":1633490210}]},"public_addr":"172.21.15.16:6819/1633490210","cluster_addr":"172.21.15.16:6821/1633490210","heartbeat_back_addr":"172.21.15.16:6825/1633490210","heartbeat_front_addr":"172.21.15.16:6823/1633490210","state":["exists","up"]},{"osd":7,"uuid":"61259d55-32c3-438a-bdac-0daf88896936","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6817","nonce":2208117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6819","nonce":2208117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6823","nonce":2208117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2208117973},{"type":"v1","addr":"172.21.15.119:6821","nonce":2208117973}]},"public_addr":"172.21.15.119:6817/2208117973","cluster_addr":"172.21.15.119:6819/2208117973","heartbeat_back_addr":"172.21.15.119:6823/2208117973","heartbeat_front_addr":"172.21.15.119:6821/2208117973","state":["exists","up"]},{"osd":8,"uuid":"59686f85-07f5-4b22-a3a6-85a29cacda2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6817","nonce":3338270245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6819","nonce":3338270245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6823","nonce":3338270245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3338270245},{"type":"v1","addr":"172.21.15.188:6821","nonce":3338270245}]},"public_addr":"172.21.15.188:6817/3338270245","cluster_addr":"172.21.15.188:6819/3338270245","heartbeat_back_addr":"172.21.15.188:6823/3338270245","heartbeat_front_addr":"172.21.15.188:6821/3338270245","state":["exists","up"]},{"osd":9,"uuid":"19ad4b60-9404-402d-bd53-bcd5a7887f35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6827","nonce":3476141905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6829","nonce":3476141905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6833","nonce":3476141905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3476141905},{"type":"v1","addr":"172.21.15.16:6831","nonce":3476141905}]},"public_addr":"172.21.15.16:6827/3476141905","cluster_addr":"172.21.15.16:6829/3476141905","heartbeat_back_addr":"172.21.15.16:6833/3476141905","heartbeat_front_addr":"172.21.15.16:6831/3476141905","state":["exists","up"]},{"osd":10,"uuid":"c665a9ff-a65b-4ee2-936d-4169ea0ac9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6825","nonce":1724485593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6827","nonce":1724485593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6831","nonce":1724485593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1724485593},{"type":"v1","addr":"172.21.15.119:6829","nonce":1724485593}]},"public_addr":"172.21.15.119:6825/1724485593","cluster_addr":"172.21.15.119:6827/1724485593","heartbeat_back_addr":"172.21.15.119:6831/1724485593","heartbeat_front_addr":"172.21.15.119:6829/1724485593","state":["exists","up"]},{"osd":11,"uuid":"235b52a6-7965-4637-a723-c5802733e041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6825","nonce":2606023375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6827","nonce":2606023375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6831","nonce":2606023375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":2606023375},{"type":"v1","addr":"172.21.15.188:6829","nonce":2606023375}]},"public_addr":"172.21.15.188:6825/2606023375","cluster_addr":"172.21.15.188:6827/2606023375","heartbeat_back_addr":"172.21.15.188:6831/2606023375","heartbeat_front_addr":"172.21.15.188:6829/2606023375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:30.795965+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:25:55.988843+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:19.208669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:26:52.230649+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:18.711617+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:27:43.062467+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:17.158693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:28:44.151607+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:08.884216+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:29:43.765880+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:11.552907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-13T03:30:37.113219+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/2756805961":"2024-04-14T03:34:45.365818+0000","172.21.15.16:0/2348781121":"2024-04-14T03:34:45.365818+0000","172.21.15.16:0/1183476525":"2024-04-14T03:34:45.365818+0000","172.21.15.16:6800/1721562410":"2024-04-14T03:34:45.365818+0000","172.21.15.16:0/2242726589":"2024-04-14T03:34:45.365818+0000","172.21.15.16:6801/1721562410":"2024-04-14T03:34:45.365818+0000","172.21.15.16:6800/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/83197818":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/20480070":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/3152772162":"2024-04-14T03:23:04.480630+0000","172.21.15.16:6801/2135228404":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/2468290259":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6801/164620940":"2024-04-14T03:23:39.714975+0000","172.21.15.16:0/1664031642":"2024-04-14T03:23:04.480630+0000","172.21.15.16:0/3312707257":"2024-04-14T03:23:39.714975+0000","172.21.15.16:6800/2135228404":"2024-04-14T03:23:04.480630+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]},{"pool":4,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-13T03:52:03.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.761+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3f180a3da0 msgr2=0x7f3f180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.761+0000 7f3f33200640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3f180a3da0 0x7f3f180a6260 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3f3419bf80 tx=0x7f3f1c005f90 comp rx=0 tx=0).stop 2024-04-13T03:52:03.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.761+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34100fb0 msgr2=0x7f3f341a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:03.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.761+0000 7f3f33200640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34100fb0 0x7f3f341a0670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3f24000f80 tx=0x7f3f24034650 comp rx=0 tx=0).stop 2024-04-13T03:52:03.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.762+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 shutdown_connections 2024-04-13T03:52:03.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.762+0000 7f3f33200640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3f180a3da0 0x7f3f180a6260 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.762+0000 7f3f33200640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f34105cf0 0x7f3f3419a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.762+0000 7f3f33200640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f34101970 0x7f3f341a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.762+0000 7f3f33200640 1 --2- 172.21.15.16:0/1970638480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f34100fb0 0x7f3f341a0670 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:03.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.762+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 >> 172.21.15.16:0/1970638480 conn(0x7f3f34076800 msgr2=0x7f3f3410c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:03.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.762+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 shutdown_connections 2024-04-13T03:52:03.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:03.762+0000 7f3f33200640 1 -- 172.21.15.16:0/1970638480 wait complete. 2024-04-13T03:52:03.783 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-13T03:52:03.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:03 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3090137366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:52:03.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:03 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2024431419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-13T03:52:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.012+0000 7f78b7200640 1 -- 172.21.15.16:0/220464088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 msgr2=0x7f78b81127c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.012+0000 7f78b7200640 1 --2- 172.21.15.16:0/220464088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 0x7f78b81127c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009930 tx=0x7f78a002bc20 comp rx=0 tx=0).stop 2024-04-13T03:52:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.012+0000 7f78b7200640 1 -- 172.21.15.16:0/220464088 shutdown_connections 2024-04-13T03:52:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.012+0000 7f78b7200640 1 --2- 172.21.15.16:0/220464088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78b8112d00 0x7f78b81150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.012+0000 7f78b7200640 1 --2- 172.21.15.16:0/220464088 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 0x7f78b81127c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.012+0000 7f78b7200640 1 --2- 172.21.15.16:0/220464088 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b80764f0 0x7f78b80768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.012+0000 7f78b7200640 1 -- 172.21.15.16:0/220464088 >> 172.21.15.16:0/220464088 conn(0x7f78b80ffb30 msgr2=0x7f78b8101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:04.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.012+0000 7f78b7200640 1 -- 172.21.15.16:0/220464088 shutdown_connections 2024-04-13T03:52:04.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.012+0000 7f78b7200640 1 -- 172.21.15.16:0/220464088 wait complete. 2024-04-13T03:52:04.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.013+0000 7f78b7200640 1 Processor -- start 2024-04-13T03:52:04.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b7200640 1 -- start start 2024-04-13T03:52:04.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78b80764f0 0x7f78b819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:04.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b7200640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 0x7f78b819f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:04.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b7200640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b8112d00 0x7f78b81a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:04.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b7200640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8118850 con 0x7f78b8112d00 2024-04-13T03:52:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b7200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f78b81186d0 con 0x7f78b80764f0 2024-04-13T03:52:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b7200640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f78b81189d0 con 0x7f78b8076e10 2024-04-13T03:52:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b8112d00 0x7f78b81a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78afe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 0x7f78b819f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78afe00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 0x7f78b819f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:33818/0 (socket says 172.21.15.16:33818) 2024-04-13T03:52:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78afe00640 1 -- 172.21.15.16:0/3628433347 learned_addr learned my addr 172.21.15.16:0/3628433347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b6800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b8112d00 0x7f78b81a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40836/0 (socket says 172.21.15.16:40836) 2024-04-13T03:52:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.014+0000 7f78b5e00640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78b80764f0 0x7f78b819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:04.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.015+0000 7f78afe00640 1 -- 172.21.15.16:0/3628433347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78b80764f0 msgr2=0x7f78b819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.015+0000 7f78afe00640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78b80764f0 0x7f78b819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.015+0000 7f78afe00640 1 -- 172.21.15.16:0/3628433347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b8112d00 msgr2=0x7f78b81a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.015+0000 7f78afe00640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b8112d00 0x7f78b81a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.015+0000 7f78afe00640 1 -- 172.21.15.16:0/3628433347 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b81a3dc0 con 0x7f78b8076e10 2024-04-13T03:52:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.015+0000 7f78afe00640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 0x7f78b819f310 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009430 tx=0x7f78a002cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.015+0000 7f78aea00640 1 -- 172.21.15.16:0/3628433347 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0041070 con 0x7f78b8076e10 2024-04-13T03:52:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.015+0000 7f78aea00640 1 -- 172.21.15.16:0/3628433347 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78a0004450 con 0x7f78b8076e10 2024-04-13T03:52:04.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.015+0000 7f78aea00640 1 -- 172.21.15.16:0/3628433347 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0003dc0 con 0x7f78b8076e10 2024-04-13T03:52:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.016+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b81a4050 con 0x7f78b8076e10 2024-04-13T03:52:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.016+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b81ab940 con 0x7f78b8076e10 2024-04-13T03:52:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.017+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b8077c30 con 0x7f78b8076e10 2024-04-13T03:52:04.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.019+0000 7f78aea00640 1 -- 172.21.15.16:0/3628433347 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f78a0044070 con 0x7f78b8076e10 2024-04-13T03:52:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.020+0000 7f78aea00640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f789c0a3da0 0x7f789c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.020+0000 7f78aea00640 1 -- 172.21.15.16:0/3628433347 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 9554+0+0 (secure 0 0 0) 0x7f78a00f2830 con 0x7f78b8076e10 2024-04-13T03:52:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.021+0000 7f78b5e00640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f789c0a3da0 0x7f789c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:04.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.021+0000 7f78b5e00640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f789c0a3da0 0x7f789c0a6260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009bd0 tx=0x7f78a8005f90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:04.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.023+0000 7f78aea00640 1 -- 172.21.15.16:0/3628433347 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78a00be520 con 0x7f78b8076e10 2024-04-13T03:52:04.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.272+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f78b810de90 con 0x7f78b8076e10 2024-04-13T03:52:04.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.284+0000 7f78aea00640 1 -- 172.21.15.16:0/3628433347 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) ==== 161+0+0 (secure 0 0 0) 0x7f78a00c2820 con 0x7f78b8076e10 2024-04-13T03:52:04.285 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-13T03:52:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f789c0a3da0 msgr2=0x7f789c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f789c0a3da0 0x7f789c0a6260 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009bd0 tx=0x7f78a8005f90 comp rx=0 tx=0).stop 2024-04-13T03:52:04.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 msgr2=0x7f78b819f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:04.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 0x7f78b819f310 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009430 tx=0x7f78a002cb10 comp rx=0 tx=0).stop 2024-04-13T03:52:04.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 shutdown_connections 2024-04-13T03:52:04.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f789c0a3da0 0x7f789c0a6260 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78b8112d00 0x7f78b81a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f78b8076e10 0x7f78b819f310 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 --2- 172.21.15.16:0/3628433347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78b80764f0 0x7f78b819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 >> 172.21.15.16:0/3628433347 conn(0x7f78b80ffb30 msgr2=0x7f78b8101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:04.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 shutdown_connections 2024-04-13T03:52:04.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.288+0000 7f78b7200640 1 -- 172.21.15.16:0/3628433347 wait complete. 2024-04-13T03:52:04.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-13T03:52:04.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.524+0000 7f8de9a00640 1 -- 172.21.15.16:0/1196606858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41071a0 msgr2=0x7f8de410ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:04.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.524+0000 7f8de9a00640 1 --2- 172.21.15.16:0/1196606858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41071a0 0x7f8de410ed70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009be0 tx=0x7f8ddc02bef0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.525+0000 7f8de9a00640 1 -- 172.21.15.16:0/1196606858 shutdown_connections 2024-04-13T03:52:04.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.525+0000 7f8de9a00640 1 --2- 172.21.15.16:0/1196606858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41071a0 0x7f8de410ed70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.525+0000 7f8de9a00640 1 --2- 172.21.15.16:0/1196606858 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de4102a80 0x7f8de4106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.525+0000 7f8de9a00640 1 --2- 172.21.15.16:0/1196606858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8de41020d0 0x7f8de41024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.525+0000 7f8de9a00640 1 -- 172.21.15.16:0/1196606858 >> 172.21.15.16:0/1196606858 conn(0x7f8de40fd990 msgr2=0x7f8de40ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:04.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.525+0000 7f8de9a00640 1 -- 172.21.15.16:0/1196606858 shutdown_connections 2024-04-13T03:52:04.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.525+0000 7f8de9a00640 1 -- 172.21.15.16:0/1196606858 wait complete. 2024-04-13T03:52:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.526+0000 7f8de9a00640 1 Processor -- start 2024-04-13T03:52:04.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.526+0000 7f8de9a00640 1 -- start start 2024-04-13T03:52:04.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.526+0000 7f8de9a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8de41020d0 0x7f8de41a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:04.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.526+0000 7f8de9a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de41071a0 0x7f8de41a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:04.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.526+0000 7f8de9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de419a760 0x7f8de419abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:04.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.526+0000 7f8de9a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8de41122b0 con 0x7f8de41020d0 2024-04-13T03:52:04.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.526+0000 7f8de9a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8de4112130 con 0x7f8de419a760 2024-04-13T03:52:04.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.526+0000 7f8de9a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f8de4112430 con 0x7f8de41071a0 2024-04-13T03:52:04.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de41071a0 0x7f8de41a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de3e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8de41020d0 0x7f8de41a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de3400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de41071a0 0x7f8de41a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:33872/0 (socket says 172.21.15.16:33872) 2024-04-13T03:52:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de3400640 1 -- 172.21.15.16:0/4179528112 learned_addr learned my addr 172.21.15.16:0/4179528112 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de9000640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de419a760 0x7f8de419abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de3400640 1 -- 172.21.15.16:0/4179528112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de419a760 msgr2=0x7f8de419abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de3400640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de419a760 0x7f8de419abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de3400640 1 -- 172.21.15.16:0/4179528112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8de41020d0 msgr2=0x7f8de41a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de3400640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8de41020d0 0x7f8de41a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.527+0000 7f8de3400640 1 -- 172.21.15.16:0/4179528112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de419b360 con 0x7f8de41071a0 2024-04-13T03:52:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.528+0000 7f8de3400640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de41071a0 0x7f8de41a0be0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400e9f0 tx=0x7f8dd400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.528+0000 7f8de0c00640 1 -- 172.21.15.16:0/4179528112 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd400ccc0 con 0x7f8de41071a0 2024-04-13T03:52:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.528+0000 7f8de0c00640 1 -- 172.21.15.16:0/4179528112 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dd4005500 con 0x7f8de41071a0 2024-04-13T03:52:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.528+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de41a7440 con 0x7f8de41071a0 2024-04-13T03:52:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.528+0000 7f8de0c00640 1 -- 172.21.15.16:0/4179528112 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4010690 con 0x7f8de41071a0 2024-04-13T03:52:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.528+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de41a7840 con 0x7f8de41071a0 2024-04-13T03:52:04.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.529+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de41035a0 con 0x7f8de41071a0 2024-04-13T03:52:04.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.531+0000 7f8de0c00640 1 -- 172.21.15.16:0/4179528112 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8dd40046c0 con 0x7f8de41071a0 2024-04-13T03:52:04.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.532+0000 7f8de0c00640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f8dc80a3da0 0x7f8dc80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.533+0000 7f8de0c00640 1 -- 172.21.15.16:0/4179528112 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9896+0+0 (secure 0 0 0) 0x7f8dd40d0ff0 con 0x7f8de41071a0 2024-04-13T03:52:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.533+0000 7f8de3e00640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f8dc80a3da0 0x7f8dc80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.534+0000 7f8de3e00640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f8dc80a3da0 0x7f8dc80a6260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc005fd0 tx=0x7f8dcc009260 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:04.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.536+0000 7f8de0c00640 1 -- 172.21.15.16:0/4179528112 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8dd409d110 con 0x7f8de41071a0 2024-04-13T03:52:04.604 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d[69557]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.604 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d[69557]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.604 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mds[69580]: -- 172.21.15.16:0/4108705550 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5590144afa40 con 0x559014483000 2024-04-13T03:52:04.604 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d[69557]: starting mds.d at 2024-04-13T03:52:04.605 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j[69190]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.605 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j[69190]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.605 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mds[69194]: -- 172.21.15.16:0/1643060375 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c1b487da40 con 0x55c1b4850800 2024-04-13T03:52:04.605 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j[69190]: starting mds.j at 2024-04-13T03:52:04.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a[31768]: 2024-04-13T03:52:04.274+0000 7f3201a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-13T03:52:04.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1970638480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:52:04.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mon[31772]: pgmap v544: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 115 KiB/s wr, 32 op/s 2024-04-13T03:52:04.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3628433347' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-13T03:52:04.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-13T03:52:04.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mon[31772]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-13T03:52:04.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mon[31772]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-13T03:52:04.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-13T03:52:04.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mon[31772]: osdmap e118: 12 total, 12 up, 12 in 2024-04-13T03:52:04.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:04 smithi016 ceph-mon[31772]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-13T03:52:04.622 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f[67114]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.622 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f[67114]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.623 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mds[67118]: -- 172.21.15.188:0/1902642591 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cd8b547a40 con 0x55cd8b51a800 2024-04-13T03:52:04.623 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f[67114]: starting mds.f at 2024-04-13T03:52:04.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1970638480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:52:04.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mon[36398]: pgmap v544: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 115 KiB/s wr, 32 op/s 2024-04-13T03:52:04.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3628433347' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-13T03:52:04.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-13T03:52:04.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mon[36398]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-13T03:52:04.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mon[36398]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-13T03:52:04.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-13T03:52:04.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mon[36398]: osdmap e118: 12 total, 12 up, 12 in 2024-04-13T03:52:04.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mon[36398]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-13T03:52:04.624 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c[67425]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.625 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c[67425]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.625 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mds[67429]: -- 172.21.15.188:0/1134576864 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b0302dfa40 con 0x55b0302b2800 2024-04-13T03:52:04.625 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c[67425]: starting mds.c at 2024-04-13T03:52:04.625 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i[67734]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.625 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i[67734]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.625 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i[67734]: starting mds.i at 2024-04-13T03:52:04.626 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l[66805]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.626 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l[66805]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.626 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-mds[66809]: -- 172.21.15.188:0/259950704 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558e759cba40 con 0x558e7599ec00 2024-04-13T03:52:04.626 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l[66805]: starting mds.l at 2024-04-13T03:52:04.626 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:52:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l[66805]: 2024-04-13T03:52:04.327+0000 7fe15d8ba700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-04-13T03:52:04.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:04.784+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f8de4102ad0 con 0x7f8de41071a0 2024-04-13T03:52:04.821 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.821 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.821 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: starting mds.b at 2024-04-13T03:52:04.822 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k[65292]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.822 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k[65292]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.822 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k[65292]: starting mds.k at 2024-04-13T03:52:04.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1970638480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-13T03:52:04.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mon[35835]: pgmap v544: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 115 KiB/s wr, 32 op/s 2024-04-13T03:52:04.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3628433347' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-13T03:52:04.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-13T03:52:04.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mon[35835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-13T03:52:04.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mon[35835]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-13T03:52:04.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-13T03:52:04.823 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mon[35835]: osdmap e118: 12 total, 12 up, 12 in 2024-04-13T03:52:04.824 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mon[35835]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-13T03:52:04.824 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-e[65609]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.824 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-e[65609]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.824 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-e[65609]: starting mds.e at 2024-04-13T03:52:04.824 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h[65914]: ignoring --setuser ceph since I am not root 2024-04-13T03:52:04.825 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h[65914]: ignoring --setgroup ceph since I am not root 2024-04-13T03:52:04.825 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-mds[65918]: -- 172.21.15.119:0/681993551 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d711c8da40 con 0x55d711c61000 2024-04-13T03:52:04.825 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:52:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h[65914]: starting mds.h at 2024-04-13T03:52:05.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.356+0000 7f8de0c00640 1 -- 172.21.15.16:0/4179528112 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) ==== 104+0+0 (secure 0 0 0) 0x7f8dd409ca10 con 0x7f8de41071a0 2024-04-13T03:52:05.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.361+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f8dc80a3da0 msgr2=0x7f8dc80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:05.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.361+0000 7f8de9a00640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f8dc80a3da0 0x7f8dc80a6260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc005fd0 tx=0x7f8dcc009260 comp rx=0 tx=0).stop 2024-04-13T03:52:05.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.361+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de41071a0 msgr2=0x7f8de41a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:05.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.361+0000 7f8de9a00640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de41071a0 0x7f8de41a0be0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400e9f0 tx=0x7f8dd400edf0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.362+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 shutdown_connections 2024-04-13T03:52:05.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.362+0000 7f8de9a00640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f8dc80a3da0 0x7f8dc80a6260 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.362+0000 7f8de9a00640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de419a760 0x7f8de419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.362+0000 7f8de9a00640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8de41071a0 0x7f8de41a0be0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.362+0000 7f8de9a00640 1 --2- 172.21.15.16:0/4179528112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8de41020d0 0x7f8de41a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.362+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 >> 172.21.15.16:0/4179528112 conn(0x7f8de40fd990 msgr2=0x7f8de40fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:05.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.362+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 shutdown_connections 2024-04-13T03:52:05.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.363+0000 7f8de9a00640 1 -- 172.21.15.16:0/4179528112 wait complete. 2024-04-13T03:52:05.383 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-13T03:52:05.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.582+0000 7f9107e00640 1 -- 172.21.15.16:0/1000385168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9108102a80 msgr2=0x7f9108106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.582+0000 7f9107e00640 1 --2- 172.21.15.16:0/1000385168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9108102a80 0x7f9108106ac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009860 tx=0x7f90f402bb70 comp rx=0 tx=0).stop 2024-04-13T03:52:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.583+0000 7f9107e00640 1 -- 172.21.15.16:0/1000385168 shutdown_connections 2024-04-13T03:52:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.583+0000 7f9107e00640 1 --2- 172.21.15.16:0/1000385168 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91081071a0 0x7f910810ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.583+0000 7f9107e00640 1 --2- 172.21.15.16:0/1000385168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9108102a80 0x7f9108106ac0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.583+0000 7f9107e00640 1 --2- 172.21.15.16:0/1000385168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91081020d0 0x7f91081024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.583+0000 7f9107e00640 1 -- 172.21.15.16:0/1000385168 >> 172.21.15.16:0/1000385168 conn(0x7f91080fd990 msgr2=0x7f91080ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:05.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.583+0000 7f9107e00640 1 -- 172.21.15.16:0/1000385168 shutdown_connections 2024-04-13T03:52:05.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.583+0000 7f9107e00640 1 -- 172.21.15.16:0/1000385168 wait complete. 2024-04-13T03:52:05.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.584+0000 7f9107e00640 1 Processor -- start 2024-04-13T03:52:05.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.584+0000 7f9107e00640 1 -- start start 2024-04-13T03:52:05.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.584+0000 7f9107e00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91081020d0 0x7f91081a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:05.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.584+0000 7f9107e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91081071a0 0x7f91081a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9107e00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f910819a870 0x7f910819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9107e00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9108112230 con 0x7f91081020d0 2024-04-13T03:52:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9107e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f91081120b0 con 0x7f91081071a0 2024-04-13T03:52:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9107e00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f91081123b0 con 0x7f910819a870 2024-04-13T03:52:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9106a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91081020d0 0x7f91081a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:05.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9106000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91081071a0 0x7f91081a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9106a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91081020d0 0x7f91081a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40898/0 (socket says 172.21.15.16:40898) 2024-04-13T03:52:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9106a00640 1 -- 172.21.15.16:0/1828995964 learned_addr learned my addr 172.21.15.16:0/1828995964 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9107400640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f910819a870 0x7f910819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9106a00640 1 -- 172.21.15.16:0/1828995964 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f910819a870 msgr2=0x7f910819ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9106a00640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f910819a870 0x7f910819ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9106a00640 1 -- 172.21.15.16:0/1828995964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91081071a0 msgr2=0x7f91081a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:05.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9106a00640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91081071a0 0x7f91081a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.585+0000 7f9106a00640 1 -- 172.21.15.16:0/1828995964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910819b390 con 0x7f91081020d0 2024-04-13T03:52:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.586+0000 7f9106a00640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91081020d0 0x7f91081a06c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f90f800c420 tx=0x7f90f8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.586+0000 7f90ff400640 1 -- 172.21.15.16:0/1828995964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8003970 con 0x7f91081020d0 2024-04-13T03:52:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.586+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910819b680 con 0x7f91081020d0 2024-04-13T03:52:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.586+0000 7f90ff400640 1 -- 172.21.15.16:0/1828995964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90f8004020 con 0x7f91081020d0 2024-04-13T03:52:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.586+0000 7f90ff400640 1 -- 172.21.15.16:0/1828995964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f800bad0 con 0x7f91081020d0 2024-04-13T03:52:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.586+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91081a7920 con 0x7f91081020d0 2024-04-13T03:52:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.587+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9108068d00 con 0x7f91081020d0 2024-04-13T03:52:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.589+0000 7f90ff400640 1 -- 172.21.15.16:0/1828995964 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f90f801d020 con 0x7f91081020d0 2024-04-13T03:52:05.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.590+0000 7f90ff400640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f90ec0a3d20 0x7f90ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.591+0000 7f90ff400640 1 -- 172.21.15.16:0/1828995964 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9896+0+0 (secure 0 0 0) 0x7f90f80d0bd0 con 0x7f91081020d0 2024-04-13T03:52:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.591+0000 7f9106000640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f90ec0a3d20 0x7f90ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:05.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.592+0000 7f9106000640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f90ec0a3d20 0x7f90ec0a61e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009830 tx=0x7f90f40056d0 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:05.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.595+0000 7f90ff400640 1 -- 172.21.15.16:0/1828995964 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90f809cd10 con 0x7f91081020d0 2024-04-13T03:52:05.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:05 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/4179528112' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:52:05.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:05 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:52:05.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:05 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/4179528112' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:52:05.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:05 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:52:05.722 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:05 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/4179528112' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:52:05.722 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:05 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-13T03:52:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:05.800+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9108102a80 con 0x7f91081020d0 2024-04-13T03:52:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.366+0000 7f90ff400640 1 -- 172.21.15.16:0/1828995964 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v119) ==== 181+0+0 (secure 0 0 0) 0x7f90f809c610 con 0x7f91081020d0 2024-04-13T03:52:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-04-13T03:52:06.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f90ec0a3d20 msgr2=0x7f90ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:06.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f90ec0a3d20 0x7f90ec0a61e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009830 tx=0x7f90f40056d0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91081020d0 msgr2=0x7f91081a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:06.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91081020d0 0x7f91081a06c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f90f800c420 tx=0x7f90f8009f90 comp rx=0 tx=0).stop 2024-04-13T03:52:06.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 shutdown_connections 2024-04-13T03:52:06.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f90ec0a3d20 0x7f90ec0a61e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f910819a870 0x7f910819ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91081071a0 0x7f91081a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 --2- 172.21.15.16:0/1828995964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91081020d0 0x7f91081a06c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.370+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 >> 172.21.15.16:0/1828995964 conn(0x7f91080fd990 msgr2=0x7f91080ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.371+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 shutdown_connections 2024-04-13T03:52:06.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.371+0000 7f9107e00640 1 -- 172.21.15.16:0/1828995964 wait complete. 2024-04-13T03:52:06.398 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-13T03:52:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-13T03:52:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-13T03:52:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: Cluster is now healthy 2024-04-13T03:52:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-13T03:52:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-13T03:52:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-13T03:52:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-13T03:52:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-13T03:52:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6832/642690082,v1:172.21.15.188:6834/642690082] up:boot 2024-04-13T03:52:06.572 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-13T03:52:06.573 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6833/2374593258,v1:172.21.15.188:6835/2374593258] up:boot 2024-04-13T03:52:06.573 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6834/2299215538,v1:172.21.15.119:6835/2299215538] up:boot 2024-04-13T03:52:06.573 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-13T03:52:06.573 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6832/2219748334,v1:172.21.15.119:6833/2219748334] up:boot 2024-04-13T03:52:06.573 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6837/431590167,v1:172.21.15.119:6838/431590167] up:boot 2024-04-13T03:52:06.573 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-13T03:52:06.573 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6836/893510656,v1:172.21.15.188:6837/893510656] up:boot 2024-04-13T03:52:06.573 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.188:6838/346280999,v1:172.21.15.188:6839/346280999] up:boot 2024-04-13T03:52:06.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-13T03:52:06.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.119:6836/4173383511,v1:172.21.15.119:6839/4173383511] up:boot 2024-04-13T03:52:06.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.16:6834/1337955223,v1:172.21.15.16:6835/1337955223] up:boot 2024-04-13T03:52:06.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: mds.? [v2:172.21.15.16:6836/986323367,v1:172.21.15.16:6837/986323367] up:boot 2024-04-13T03:52:06.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: fsmap 12 up:standby 2024-04-13T03:52:06.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-13T03:52:06.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-13T03:52:06.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1828995964' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:06.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:06 smithi119 ceph-mon[35835]: pgmap v546: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 135 KiB/s wr, 36 op/s 2024-04-13T03:52:06.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-13T03:52:06.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-13T03:52:06.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: Cluster is now healthy 2024-04-13T03:52:06.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-13T03:52:06.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-13T03:52:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-13T03:52:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-13T03:52:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-13T03:52:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6832/642690082,v1:172.21.15.188:6834/642690082] up:boot 2024-04-13T03:52:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-13T03:52:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6833/2374593258,v1:172.21.15.188:6835/2374593258] up:boot 2024-04-13T03:52:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6834/2299215538,v1:172.21.15.119:6835/2299215538] up:boot 2024-04-13T03:52:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-13T03:52:06.605 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6832/2219748334,v1:172.21.15.119:6833/2219748334] up:boot 2024-04-13T03:52:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6837/431590167,v1:172.21.15.119:6838/431590167] up:boot 2024-04-13T03:52:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-13T03:52:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6836/893510656,v1:172.21.15.188:6837/893510656] up:boot 2024-04-13T03:52:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.188:6838/346280999,v1:172.21.15.188:6839/346280999] up:boot 2024-04-13T03:52:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-13T03:52:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.119:6836/4173383511,v1:172.21.15.119:6839/4173383511] up:boot 2024-04-13T03:52:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.16:6834/1337955223,v1:172.21.15.16:6835/1337955223] up:boot 2024-04-13T03:52:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: mds.? [v2:172.21.15.16:6836/986323367,v1:172.21.15.16:6837/986323367] up:boot 2024-04-13T03:52:06.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: fsmap 12 up:standby 2024-04-13T03:52:06.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-13T03:52:06.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-13T03:52:06.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1828995964' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:06.607 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:06 smithi016 ceph-mon[31772]: pgmap v546: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 135 KiB/s wr, 36 op/s 2024-04-13T03:52:06.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-13T03:52:06.622 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-13T03:52:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: Cluster is now healthy 2024-04-13T03:52:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-13T03:52:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-13T03:52:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-13T03:52:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-13T03:52:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-13T03:52:06.623 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6832/642690082,v1:172.21.15.188:6834/642690082] up:boot 2024-04-13T03:52:06.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-13T03:52:06.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6833/2374593258,v1:172.21.15.188:6835/2374593258] up:boot 2024-04-13T03:52:06.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6834/2299215538,v1:172.21.15.119:6835/2299215538] up:boot 2024-04-13T03:52:06.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-13T03:52:06.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6832/2219748334,v1:172.21.15.119:6833/2219748334] up:boot 2024-04-13T03:52:06.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6837/431590167,v1:172.21.15.119:6838/431590167] up:boot 2024-04-13T03:52:06.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-13T03:52:06.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6836/893510656,v1:172.21.15.188:6837/893510656] up:boot 2024-04-13T03:52:06.624 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.188:6838/346280999,v1:172.21.15.188:6839/346280999] up:boot 2024-04-13T03:52:06.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-13T03:52:06.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.119:6836/4173383511,v1:172.21.15.119:6839/4173383511] up:boot 2024-04-13T03:52:06.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.16:6834/1337955223,v1:172.21.15.16:6835/1337955223] up:boot 2024-04-13T03:52:06.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: mds.? [v2:172.21.15.16:6836/986323367,v1:172.21.15.16:6837/986323367] up:boot 2024-04-13T03:52:06.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: fsmap 12 up:standby 2024-04-13T03:52:06.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-13T03:52:06.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='mgr.25450 172.21.15.16:0/3823677576' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-13T03:52:06.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1828995964' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:06.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:06 smithi188 ceph-mon[36398]: pgmap v546: 129 pgs: 129 active+clean; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 135 KiB/s wr, 36 op/s 2024-04-13T03:52:06.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.644+0000 7fd41aa00640 1 -- 172.21.15.16:0/1527622945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd414103b60 msgr2=0x7fd41410c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:06.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.644+0000 7fd41aa00640 1 --2- 172.21.15.16:0/1527622945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd414103b60 0x7fd41410c040 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc009900 tx=0x7fd3fc02bd70 comp rx=0 tx=0).stop 2024-04-13T03:52:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.645+0000 7fd41aa00640 1 -- 172.21.15.16:0/1527622945 shutdown_connections 2024-04-13T03:52:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.645+0000 7fd41aa00640 1 --2- 172.21.15.16:0/1527622945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd41410c580 0x7fd41410ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.645+0000 7fd41aa00640 1 --2- 172.21.15.16:0/1527622945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd414103b60 0x7fd41410c040 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.645+0000 7fd41aa00640 1 --2- 172.21.15.16:0/1527622945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd414103240 0x7fd414103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.645+0000 7fd41aa00640 1 -- 172.21.15.16:0/1527622945 >> 172.21.15.16:0/1527622945 conn(0x7fd414077fc0 msgr2=0x7fd4140783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.645+0000 7fd41aa00640 1 -- 172.21.15.16:0/1527622945 shutdown_connections 2024-04-13T03:52:06.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.645+0000 7fd41aa00640 1 -- 172.21.15.16:0/1527622945 wait complete. 2024-04-13T03:52:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.646+0000 7fd41aa00640 1 Processor -- start 2024-04-13T03:52:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.646+0000 7fd41aa00640 1 -- start start 2024-04-13T03:52:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.646+0000 7fd41aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd414103240 0x7fd414100e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.646+0000 7fd41aa00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd414103b60 0x7fd4141013d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.646+0000 7fd41aa00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd41410c580 0x7fd414101910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.646+0000 7fd41aa00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd414112390 con 0x7fd41410c580 2024-04-13T03:52:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd41aa00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd414112210 con 0x7fd414103240 2024-04-13T03:52:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd41aa00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7fd414112510 con 0x7fd414103b60 2024-04-13T03:52:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd418a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd41410c580 0x7fd414101910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd413e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd414103240 0x7fd414100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd40b400640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd414103b60 0x7fd4141013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd413e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd414103240 0x7fd414100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.16:37938/0 (socket says 172.21.15.16:37938) 2024-04-13T03:52:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd413e00640 1 -- 172.21.15.16:0/2132874969 learned_addr learned my addr 172.21.15.16:0/2132874969 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd40b400640 1 -- 172.21.15.16:0/2132874969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd414103240 msgr2=0x7fd414100e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd40b400640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd414103240 0x7fd414100e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd40b400640 1 -- 172.21.15.16:0/2132874969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd41410c580 msgr2=0x7fd414101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd40b400640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd41410c580 0x7fd414101910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.647+0000 7fd40b400640 1 -- 172.21.15.16:0/2132874969 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41419ca30 con 0x7fd414103b60 2024-04-13T03:52:06.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.648+0000 7fd40b400640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd414103b60 0x7fd4141013d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc009230 tx=0x7fd3fc02dcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.648+0000 7fd411600640 1 -- 172.21.15.16:0/2132874969 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc043020 con 0x7fd414103b60 2024-04-13T03:52:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.648+0000 7fd411600640 1 -- 172.21.15.16:0/2132874969 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3fc0054e0 con 0x7fd414103b60 2024-04-13T03:52:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.648+0000 7fd411600640 1 -- 172.21.15.16:0/2132874969 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc003c90 con 0x7fd414103b60 2024-04-13T03:52:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.648+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41419ccc0 con 0x7fd414103b60 2024-04-13T03:52:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.648+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41419d230 con 0x7fd414103b60 2024-04-13T03:52:06.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.649+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd414069950 con 0x7fd414103b60 2024-04-13T03:52:06.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.652+0000 7fd411600640 1 -- 172.21.15.16:0/2132874969 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd3fc046020 con 0x7fd414103b60 2024-04-13T03:52:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.653+0000 7fd411600640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd3f80a3da0 0x7fd3f80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:06.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.653+0000 7fd411600640 1 -- 172.21.15.16:0/2132874969 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 9342+0+0 (secure 0 0 0) 0x7fd3fc0f2a80 con 0x7fd414103b60 2024-04-13T03:52:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.653+0000 7fd413e00640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd3f80a3da0 0x7fd3f80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:06.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.654+0000 7fd413e00640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd3f80a3da0 0x7fd3f80a6260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd404009fb0 tx=0x7fd404009500 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:06.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.657+0000 7fd411600640 1 -- 172.21.15.16:0/2132874969 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3fc0f4e20 con 0x7fd414103b60 2024-04-13T03:52:06.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:06.934+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd414068de0 con 0x7fd414103b60 2024-04-13T03:52:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.373+0000 7fd411600640 1 -- 172.21.15.16:0/2132874969 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v120) ==== 169+0+0 (secure 0 0 0) 0x7fd3fc04d020 con 0x7fd414103b60 2024-04-13T03:52:07.374 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-04-13T03:52:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.376+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd3f80a3da0 msgr2=0x7fd3f80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.376+0000 7fd41aa00640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd3f80a3da0 0x7fd3f80a6260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd404009fb0 tx=0x7fd404009500 comp rx=0 tx=0).stop 2024-04-13T03:52:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.376+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd414103b60 msgr2=0x7fd4141013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.376+0000 7fd41aa00640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd414103b60 0x7fd4141013d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc009230 tx=0x7fd3fc02dcf0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.377+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 shutdown_connections 2024-04-13T03:52:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.377+0000 7fd41aa00640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7fd3f80a3da0 0x7fd3f80a6260 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.377+0000 7fd41aa00640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd41410c580 0x7fd414101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.377+0000 7fd41aa00640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd414103b60 0x7fd4141013d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.377+0000 7fd41aa00640 1 --2- 172.21.15.16:0/2132874969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd414103240 0x7fd414100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.377+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 >> 172.21.15.16:0/2132874969 conn(0x7fd414077fc0 msgr2=0x7fd4140fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.377+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 shutdown_connections 2024-04-13T03:52:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.377+0000 7fd41aa00640 1 -- 172.21.15.16:0/2132874969 wait complete. 2024-04-13T03:52:07.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-13T03:52:07.573 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:07 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/1828995964' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-13T03:52:07.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:07 smithi119 ceph-mon[35835]: osdmap e119: 12 total, 12 up, 12 in 2024-04-13T03:52:07.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:07 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/2132874969' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:07.574 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:07 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.621+0000 7f3a87a00640 1 -- 172.21.15.16:0/2596780297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a80101740 msgr2=0x7f3a8010e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.621+0000 7f3a87a00640 1 --2- 172.21.15.16:0/2596780297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a80101740 0x7f3a8010e490 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009900 tx=0x7f3a6c02bc30 comp rx=0 tx=0).stop 2024-04-13T03:52:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.621+0000 7f3a87a00640 1 -- 172.21.15.16:0/2596780297 shutdown_connections 2024-04-13T03:52:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.621+0000 7f3a87a00640 1 --2- 172.21.15.16:0/2596780297 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a8010e9d0 0x7f3a80110dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.621+0000 7f3a87a00640 1 --2- 172.21.15.16:0/2596780297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a80101740 0x7f3a8010e490 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.621+0000 7f3a87a00640 1 --2- 172.21.15.16:0/2596780297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a80100e20 0x7f3a80101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.621+0000 7f3a87a00640 1 -- 172.21.15.16:0/2596780297 >> 172.21.15.16:0/2596780297 conn(0x7f3a80078020 msgr2=0x7f3a80078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:07.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.621+0000 7f3a87a00640 1 -- 172.21.15.16:0/2596780297 shutdown_connections 2024-04-13T03:52:07.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.621+0000 7f3a87a00640 1 -- 172.21.15.16:0/2596780297 wait complete. 2024-04-13T03:52:07.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.623+0000 7f3a87a00640 1 Processor -- start 2024-04-13T03:52:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.623+0000 7f3a87a00640 1 -- start start 2024-04-13T03:52:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.623+0000 7f3a87a00640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a80100e20 0x7f3a801a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.623+0000 7f3a87a00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a80101740 0x7f3a801a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.623+0000 7f3a87a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8010e9d0 0x7f3a8019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.623+0000 7f3a87a00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a80114610 con 0x7f3a80101740 2024-04-13T03:52:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.623+0000 7f3a87a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3a80114490 con 0x7f3a8010e9d0 2024-04-13T03:52:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.623+0000 7f3a87a00640 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 -- 0x7f3a80114790 con 0x7f3a80100e20 2024-04-13T03:52:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a80100e20 0x7f3a801a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8010e9d0 0x7f3a8019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85000640 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a80100e20 0x7f3a801a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.16:33938/0 (socket says 172.21.15.16:33938) 2024-04-13T03:52:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a7fe00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a80101740 0x7f3a801a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85000640 1 -- 172.21.15.16:0/3115498718 learned_addr learned my addr 172.21.15.16:0/3115498718 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-13T03:52:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85000640 1 -- 172.21.15.16:0/3115498718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8010e9d0 msgr2=0x7f3a8019aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85000640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8010e9d0 0x7f3a8019aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85000640 1 -- 172.21.15.16:0/3115498718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a80101740 msgr2=0x7f3a801a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85000640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a80101740 0x7f3a801a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85000640 1 -- 172.21.15.16:0/3115498718 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a8019b190 con 0x7f3a80100e20 2024-04-13T03:52:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85a00640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8010e9d0 0x7f3a8019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-13T03:52:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.624+0000 7f3a85000640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a80100e20 0x7f3a801a0810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400e9f0 tx=0x7f3a7400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.625+0000 7f3a7d600640 1 -- 172.21.15.16:0/3115498718 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7400cd80 con 0x7f3a80100e20 2024-04-13T03:52:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.625+0000 7f3a7d600640 1 -- 172.21.15.16:0/3115498718 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a74010910 con 0x7f3a80100e20 2024-04-13T03:52:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.625+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a8019b480 con 0x7f3a80100e20 2024-04-13T03:52:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.625+0000 7f3a7d600640 1 -- 172.21.15.16:0/3115498718 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74018990 con 0x7f3a80100e20 2024-04-13T03:52:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.625+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a801a76d0 con 0x7f3a80100e20 2024-04-13T03:52:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.627+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a80068550 con 0x7f3a80100e20 2024-04-13T03:52:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.628+0000 7f3a7d600640 1 -- 172.21.15.16:0/3115498718 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3a74010ac0 con 0x7f3a80100e20 2024-04-13T03:52:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.629+0000 7f3a7d600640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3a680a3d20 0x7f3a680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:52:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.629+0000 7f3a7d600640 1 -- 172.21.15.16:0/3115498718 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8754+0+0 (secure 0 0 0) 0x7f3a740d0c80 con 0x7f3a80100e20 2024-04-13T03:52:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.630+0000 7f3a7fe00640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3a680a3d20 0x7f3a680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:52:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.631+0000 7f3a7fe00640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3a680a3d20 0x7f3a680a61e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c000c00 tx=0x7f3a6c02df90 comp rx=0 tx=0).ready entity=mgr.25450 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:52:07.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.634+0000 7f3a7d600640 1 -- 172.21.15.16:0/3115498718 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a7409cb80 con 0x7f3a80100e20 2024-04-13T03:52:07.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:07.840+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3a80109b60 con 0x7f3a80100e20 2024-04-13T03:52:07.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:07 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/1828995964' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-13T03:52:07.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:07 smithi016 ceph-mon[31772]: osdmap e119: 12 total, 12 up, 12 in 2024-04-13T03:52:07.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:07 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/2132874969' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:07.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:07 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:07.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:07 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/1828995964' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-13T03:52:07.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:07 smithi188 ceph-mon[36398]: osdmap e119: 12 total, 12 up, 12 in 2024-04-13T03:52:07.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:07 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/2132874969' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:07.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:07 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:08.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.386+0000 7f3a7d600640 1 -- 172.21.15.16:0/3115498718 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v121) ==== 178+0+0 (secure 0 0 0) 0x7f3a740a0e80 con 0x7f3a80100e20 2024-04-13T03:52:08.398 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' removed 2024-04-13T03:52:08.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.392+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3a680a3d20 msgr2=0x7f3a680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:08.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.392+0000 7f3a87a00640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3a680a3d20 0x7f3a680a61e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c000c00 tx=0x7f3a6c02df90 comp rx=0 tx=0).stop 2024-04-13T03:52:08.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.392+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a80100e20 msgr2=0x7f3a801a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-13T03:52:08.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.392+0000 7f3a87a00640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a80100e20 0x7f3a801a0810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400e9f0 tx=0x7f3a7400eeb0 comp rx=0 tx=0).stop 2024-04-13T03:52:08.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.393+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 shutdown_connections 2024-04-13T03:52:08.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.393+0000 7f3a87a00640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.16:6800/1691605870,v1:172.21.15.16:6801/1691605870] conn(0x7f3a680a3d20 0x7f3a680a61e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:08.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.393+0000 7f3a87a00640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8010e9d0 0x7f3a8019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:08.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.393+0000 7f3a87a00640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a80101740 0x7f3a801a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:08.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.393+0000 7f3a87a00640 1 --2- 172.21.15.16:0/3115498718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3a80100e20 0x7f3a801a0810 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:52:08.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.393+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 >> 172.21.15.16:0/3115498718 conn(0x7f3a80078020 msgr2=0x7f3a8010ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-13T03:52:08.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.393+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 shutdown_connections 2024-04-13T03:52:08.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-13T03:52:08.393+0000 7f3a87a00640 1 -- 172.21.15.16:0/3115498718 wait complete. 2024-04-13T03:52:08.432 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-13T03:52:08.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-13T03:52:08.602 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-13T03:52:08.632 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-13T03:52:08.633 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-13T03:52:08.675 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-13T03:52:08.690 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-13T03:52:08.705 INFO:tasks.cephadm:Stopping all daemons... 2024-04-13T03:52:08.706 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-13T03:52:08.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.a 2024-04-13T03:52:08.792 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:08 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-13T03:52:08.792 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:08 smithi016 ceph-mon[31772]: osdmap e120: 12 total, 12 up, 12 in 2024-04-13T03:52:08.792 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:08 smithi016 ceph-mon[31772]: from='client.? 172.21.15.16:0/3115498718' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:08.792 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:08 smithi016 ceph-mon[31772]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:08.793 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:08 smithi016 ceph-mon[31772]: pgmap v549: 33 pgs: 33 active+clean; 307 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:08.820 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:08 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-13T03:52:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:08 smithi119 ceph-mon[35835]: osdmap e120: 12 total, 12 up, 12 in 2024-04-13T03:52:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:08 smithi119 ceph-mon[35835]: from='client.? 172.21.15.16:0/3115498718' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:08 smithi119 ceph-mon[35835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:08.821 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:08 smithi119 ceph-mon[35835]: pgmap v549: 33 pgs: 33 active+clean; 307 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:08.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:08 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-13T03:52:08.872 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:08 smithi188 ceph-mon[36398]: osdmap e120: 12 total, 12 up, 12 in 2024-04-13T03:52:08.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:08 smithi188 ceph-mon[36398]: from='client.? 172.21.15.16:0/3115498718' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:08.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:08 smithi188 ceph-mon[36398]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-13T03:52:08.873 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:08 smithi188 ceph-mon[36398]: pgmap v549: 33 pgs: 33 active+clean; 307 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:09.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:08 smithi016 systemd[1]: Stopping Ceph mon.a for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:09.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:09 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a[31768]: 2024-04-13T03:52:09.286+0000 7f3208800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:52:09.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:09 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a[31768]: 2024-04-13T03:52:09.286+0000 7f3208800700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-13T03:52:09.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:09 smithi016 podman[91279]: 2024-04-13 03:52:09.525997324 +0000 UTC m=+0.363579984 container died c421eb666bd809ce4d8d64dd3171b5653e89a67954ff15c15e7e2ec85a86fa4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-13T03:52:10.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:10 smithi016 podman[91279]: 2024-04-13 03:52:10.386027757 +0000 UTC m=+1.223610450 container cleanup c421eb666bd809ce4d8d64dd3171b5653e89a67954ff15c15e7e2ec85a86fa4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-13T03:52:10.604 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:10 smithi016 bash[91279]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a 2024-04-13T03:52:11.104 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 13 03:52:10 smithi016 podman[91293]: 2024-04-13 03:52:10.679059815 +0000 UTC m=+1.136430432 container remove c421eb666bd809ce4d8d64dd3171b5653e89a67954ff15c15e7e2ec85a86fa4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-13T03:52:11.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.a.service' 2024-04-13T03:52:11.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:11.319 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-13T03:52:11.319 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-13T03:52:11.319 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.b 2024-04-13T03:52:11.693 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:11 smithi119 systemd[1]: Stopping Ceph mon.b for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:12.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:11 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-b[35831]: 2024-04-13T03:52:11.693+0000 7fd23642f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:52:12.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:11 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-b[35831]: 2024-04-13T03:52:11.693+0000 7fd23642f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-13T03:52:12.071 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:11 smithi119 podman[73577]: 2024-04-13 03:52:11.938269821 +0000 UTC m=+0.388902865 container died a9319192e0cf8b6fa6cec5c530f78935af16ab4e192371561c81ece95bb7c572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-b, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-13T03:52:12.570 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:12 smithi119 podman[73577]: 2024-04-13 03:52:12.354692588 +0000 UTC m=+0.805325617 container cleanup a9319192e0cf8b6fa6cec5c530f78935af16ab4e192371561c81ece95bb7c572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-13T03:52:12.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 13 03:52:12 smithi119 bash[73577]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-b 2024-04-13T03:52:12.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.b.service' 2024-04-13T03:52:12.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:12.860 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-13T03:52:12.860 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-13T03:52:12.860 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.c 2024-04-13T03:52:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:12 smithi188 systemd[1]: Stopping Ceph mon.c for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:13.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:13 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-c[36381]: 2024-04-13T03:52:13.164+0000 7f03b0e34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:52:13.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:13 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-c[36381]: 2024-04-13T03:52:13.164+0000 7f03b0e34700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-13T03:52:13.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:13 smithi188 podman[75905]: 2024-04-13 03:52:13.420383134 +0000 UTC m=+0.368437944 container died 3b30980af288df4891c0c16fb7c21d0a963f3b094badaadf0f652cbfc3a4323b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-13T03:52:14.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:13 smithi188 podman[75905]: 2024-04-13 03:52:13.752887169 +0000 UTC m=+0.700941981 container cleanup 3b30980af288df4891c0c16fb7c21d0a963f3b094badaadf0f652cbfc3a4323b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True) 2024-04-13T03:52:14.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:13 smithi188 bash[75905]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-c 2024-04-13T03:52:14.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Apr 13 03:52:13 smithi188 podman[75919]: 2024-04-13 03:52:13.975510235 +0000 UTC m=+0.550293642 container remove 3b30980af288df4891c0c16fb7c21d0a963f3b094badaadf0f652cbfc3a4323b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-13T03:52:14.225 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mon.c.service' 2024-04-13T03:52:14.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:14.342 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-13T03:52:14.342 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-13T03:52:14.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.x 2024-04-13T03:52:14.853 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:52:14 smithi016 systemd[1]: Stopping Ceph mgr.x for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:15.354 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:52:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:52:14.980+0000 7fee3be00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:52:15.355 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:52:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x[32010]: 2024-04-13T03:52:14.980+0000 7fee3be00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-13T03:52:15.355 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:52:15 smithi016 podman[91401]: 2024-04-13 03:52:15.07396855 +0000 UTC m=+0.228793156 container died d7830156b6df1fb06254e00795f1f6eb7e549a96785ebe2f0d33c501db2d9d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-13T03:52:15.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:52:15 smithi016 podman[91401]: 2024-04-13 03:52:15.63294231 +0000 UTC m=+0.787766904 container cleanup d7830156b6df1fb06254e00795f1f6eb7e549a96785ebe2f0d33c501db2d9d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-13T03:52:15.854 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:52:15 smithi016 bash[91401]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x 2024-04-13T03:52:16.351 INFO:journalctl@ceph.mgr.x.smithi016.stdout:Apr 13 03:52:15 smithi016 podman[91415]: 2024-04-13 03:52:15.901286069 +0000 UTC m=+0.806743128 container remove d7830156b6df1fb06254e00795f1f6eb7e549a96785ebe2f0d33c501db2d9d3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:52:16.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.x.service' 2024-04-13T03:52:16.523 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:16.523 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-13T03:52:16.523 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-13T03:52:16.524 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.y 2024-04-13T03:52:16.820 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:52:16 smithi119 systemd[1]: Stopping Ceph mgr.y for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:17.128 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:52:16 smithi119 podman[73699]: 2024-04-13 03:52:16.830998178 +0000 UTC m=+0.114917005 container died 9069651ddd42eb3f663903251e02570c451dcf5ecf8726968eb0453cfa57ce99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:52:17.511 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:52:17 smithi119 podman[73699]: 2024-04-13 03:52:17.218579104 +0000 UTC m=+0.502497923 container cleanup 9069651ddd42eb3f663903251e02570c451dcf5ecf8726968eb0453cfa57ce99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-13T03:52:17.512 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 13 03:52:17 smithi119 bash[73699]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-y 2024-04-13T03:52:17.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.y.service' 2024-04-13T03:52:17.679 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:17.679 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-13T03:52:17.680 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-13T03:52:17.680 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.z 2024-04-13T03:52:17.990 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:52:17 smithi188 systemd[1]: Stopping Ceph mgr.z for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:18.294 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:52:17 smithi188 podman[76027]: 2024-04-13 03:52:17.990625785 +0000 UTC m=+0.118274646 container died 6df30516e333a4c4372033ca7f395464d5552c65be2760891ad0d35e80c68cde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-13T03:52:18.622 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:52:18 smithi188 podman[76027]: 2024-04-13 03:52:18.294975128 +0000 UTC m=+0.422623985 container cleanup 6df30516e333a4c4372033ca7f395464d5552c65be2760891ad0d35e80c68cde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-13T03:52:18.622 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:52:18 smithi188 bash[76027]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z 2024-04-13T03:52:18.623 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:52:18 smithi188 podman[76041]: 2024-04-13 03:52:18.513411812 +0000 UTC m=+0.519881717 container remove 6df30516e333a4c4372033ca7f395464d5552c65be2760891ad0d35e80c68cde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-13T03:52:18.623 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Apr 13 03:52:18 smithi188 systemd[1]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-13T03:52:18.715 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mgr.z.service' 2024-04-13T03:52:18.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:18.803 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-13T03:52:18.803 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-13T03:52:18.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.0 2024-04-13T03:52:19.354 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:18 smithi016 systemd[1]: Stopping Ceph osd.0 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:19.854 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:19 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0[43835]: 2024-04-13T03:52:19.460+0000 7f344a600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:52:19.854 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:19 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0[43835]: 2024-04-13T03:52:19.460+0000 7f344a600700 -1 osd.0 121 *** Got signal Terminated *** 2024-04-13T03:52:19.855 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:19 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0[43835]: 2024-04-13T03:52:19.460+0000 7f344a600700 -1 osd.0 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:52:25.354 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:25 smithi016 podman[91520]: 2024-04-13 03:52:25.022590656 +0000 UTC m=+5.721521564 container stop bafebf2370f78cd97c77d2abaaf13d4ff4fa435fbea101ba2439b4361f005275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-13T03:52:25.354 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:25 smithi016 podman[91520]: 2024-04-13 03:52:25.050227818 +0000 UTC m=+5.749158661 container died bafebf2370f78cd97c77d2abaaf13d4ff4fa435fbea101ba2439b4361f005275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-13T03:52:25.798 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:25 smithi016 podman[91520]: 2024-04-13 03:52:25.538203482 +0000 UTC m=+6.237134356 container cleanup bafebf2370f78cd97c77d2abaaf13d4ff4fa435fbea101ba2439b4361f005275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0) 2024-04-13T03:52:25.798 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:25 smithi016 bash[91520]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0 2024-04-13T03:52:26.103 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:25 smithi016 podman[91549]: 2024-04-13 03:52:25.797457082 +0000 UTC m=+0.749071085 container remove bafebf2370f78cd97c77d2abaaf13d4ff4fa435fbea101ba2439b4361f005275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:52:27.028 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:26 smithi016 podman[91638]: 2024-04-13 03:52:26.551376561 +0000 UTC m=+0.035493759 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:52:27.028 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:26 smithi016 podman[91638]: 2024-04-13 03:52:26.706831702 +0000 UTC m=+0.190948896 container create dcce1fe43bb479c1bf94eee5d93476fa031e74a9b1ecc37cb5c2fa25791677bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-13T03:52:27.028 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:27 smithi016 podman[91638]: 2024-04-13 03:52:27.027663634 +0000 UTC m=+0.511780901 container init dcce1fe43bb479c1bf94eee5d93476fa031e74a9b1ecc37cb5c2fa25791677bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:52:27.354 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:27 smithi016 podman[91638]: 2024-04-13 03:52:27.040338819 +0000 UTC m=+0.524456013 container start dcce1fe43bb479c1bf94eee5d93476fa031e74a9b1ecc37cb5c2fa25791677bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-13T03:52:27.354 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:27 smithi016 podman[91638]: 2024-04-13 03:52:27.151347747 +0000 UTC m=+0.635464970 container attach dcce1fe43bb479c1bf94eee5d93476fa031e74a9b1ecc37cb5c2fa25791677bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=) 2024-04-13T03:52:27.854 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 13 03:52:27 smithi016 podman[91638]: 2024-04-13 03:52:27.509423477 +0000 UTC m=+0.993540706 container died dcce1fe43bb479c1bf94eee5d93476fa031e74a9b1ecc37cb5c2fa25791677bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-13T03:52:28.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.0.service' 2024-04-13T03:52:28.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:28.469 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-13T03:52:28.469 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-13T03:52:28.469 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.1 2024-04-13T03:52:28.781 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:28 smithi119 systemd[1]: Stopping Ceph osd.1 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:29.070 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:28 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1[40404]: 2024-04-13T03:52:28.781+0000 7fdf82b53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:52:29.071 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:28 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1[40404]: 2024-04-13T03:52:28.781+0000 7fdf82b53700 -1 osd.1 121 *** Got signal Terminated *** 2024-04-13T03:52:29.071 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:28 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1[40404]: 2024-04-13T03:52:28.781+0000 7fdf82b53700 -1 osd.1 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:52:34.492 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:34 smithi119 podman[73822]: 2024-04-13 03:52:34.1712941 +0000 UTC m=+5.516950674 container died 8891e9f2484b22b17a6e776b41a0d44f66f81c7d6eeafc467ba9d40c8ff90d13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-13T03:52:34.821 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:34 smithi119 podman[73822]: 2024-04-13 03:52:34.541060303 +0000 UTC m=+5.886716862 container cleanup 8891e9f2484b22b17a6e776b41a0d44f66f81c7d6eeafc467ba9d40c8ff90d13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-13T03:52:34.821 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:34 smithi119 bash[73822]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1 2024-04-13T03:52:35.162 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:34 smithi119 podman[73840]: 2024-04-13 03:52:34.860112713 +0000 UTC m=+0.682963760 container remove 8891e9f2484b22b17a6e776b41a0d44f66f81c7d6eeafc467ba9d40c8ff90d13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:52:35.456 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:35 smithi119 podman[73932]: 2024-04-13 03:52:35.065502074 +0000 UTC m=+0.017207156 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:52:35.456 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:35 smithi119 podman[73932]: 2024-04-13 03:52:35.20271469 +0000 UTC m=+0.154419774 container create 2eeb434ddbc4b73f29ea28669a6923f968b804ff87ab2608d4abc799430f6c7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:52:35.456 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:35 smithi119 podman[73932]: 2024-04-13 03:52:35.456748916 +0000 UTC m=+0.408454000 container init 2eeb434ddbc4b73f29ea28669a6923f968b804ff87ab2608d4abc799430f6c7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-13T03:52:35.820 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:35 smithi119 podman[73932]: 2024-04-13 03:52:35.459965912 +0000 UTC m=+0.411670996 container start 2eeb434ddbc4b73f29ea28669a6923f968b804ff87ab2608d4abc799430f6c7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-13T03:52:35.821 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:35 smithi119 podman[73932]: 2024-04-13 03:52:35.564595223 +0000 UTC m=+0.516300316 container attach 2eeb434ddbc4b73f29ea28669a6923f968b804ff87ab2608d4abc799430f6c7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-13T03:52:36.158 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 13 03:52:35 smithi119 podman[73932]: 2024-04-13 03:52:35.852597262 +0000 UTC m=+0.804302351 container died 2eeb434ddbc4b73f29ea28669a6923f968b804ff87ab2608d4abc799430f6c7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-13T03:52:36.535 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.1.service' 2024-04-13T03:52:36.580 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:36.580 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-13T03:52:36.580 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-13T03:52:36.580 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.2 2024-04-13T03:52:36.872 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:36 smithi188 systemd[1]: Stopping Ceph osd.2 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:37.132 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:36 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2[41520]: 2024-04-13T03:52:36.904+0000 7f6a5107e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:52:37.132 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:36 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2[41520]: 2024-04-13T03:52:36.904+0000 7f6a5107e700 -1 osd.2 121 *** Got signal Terminated *** 2024-04-13T03:52:37.132 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:36 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2[41520]: 2024-04-13T03:52:36.904+0000 7f6a5107e700 -1 osd.2 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:52:42.706 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:42 smithi188 podman[76149]: 2024-04-13 03:52:42.446815586 +0000 UTC m=+5.666906871 container died 6f6efe624c0424fbe40faecc68c3302e5f76595cf117974d7e373bdddd20971a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-13T03:52:43.109 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:42 smithi188 podman[76149]: 2024-04-13 03:52:42.797089164 +0000 UTC m=+6.017180443 container cleanup 6f6efe624c0424fbe40faecc68c3302e5f76595cf117974d7e373bdddd20971a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-13T03:52:43.109 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:42 smithi188 bash[76149]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2 2024-04-13T03:52:43.372 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:43 smithi188 podman[76178]: 2024-04-13 03:52:43.10956996 +0000 UTC m=+0.658821531 container remove 6f6efe624c0424fbe40faecc68c3302e5f76595cf117974d7e373bdddd20971a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-13T03:52:43.686 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:43 smithi188 podman[76268]: 2024-04-13 03:52:43.312819056 +0000 UTC m=+0.017913167 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:52:43.686 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:43 smithi188 podman[76268]: 2024-04-13 03:52:43.416505503 +0000 UTC m=+0.121599652 container create dc739e150e82cf3ecdcb1a9e339b4a66edd2dcaa3ceeae154a8d1bfccd17469b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:52:43.981 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:43 smithi188 podman[76268]: 2024-04-13 03:52:43.686653103 +0000 UTC m=+0.391747222 container init dc739e150e82cf3ecdcb1a9e339b4a66edd2dcaa3ceeae154a8d1bfccd17469b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-13T03:52:43.981 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:43 smithi188 podman[76268]: 2024-04-13 03:52:43.690309251 +0000 UTC m=+0.395403361 container start dc739e150e82cf3ecdcb1a9e339b4a66edd2dcaa3ceeae154a8d1bfccd17469b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:52:43.981 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:43 smithi188 podman[76268]: 2024-04-13 03:52:43.777802184 +0000 UTC m=+0.482896309 container attach dc739e150e82cf3ecdcb1a9e339b4a66edd2dcaa3ceeae154a8d1bfccd17469b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-13T03:52:44.372 INFO:journalctl@ceph.osd.2.smithi188.stdout:Apr 13 03:52:43 smithi188 podman[76268]: 2024-04-13 03:52:43.981404558 +0000 UTC m=+0.686498675 container died dc739e150e82cf3ecdcb1a9e339b4a66edd2dcaa3ceeae154a8d1bfccd17469b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-2-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-13T03:52:44.534 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.2.service' 2024-04-13T03:52:44.578 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:44.578 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-13T03:52:44.578 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-13T03:52:44.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.3 2024-04-13T03:52:45.104 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:44 smithi016 systemd[1]: Stopping Ceph osd.3 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:45.604 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3[50416]: 2024-04-13T03:52:45.197+0000 7f92dda00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:52:45.604 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3[50416]: 2024-04-13T03:52:45.197+0000 7f92dda00700 -1 osd.3 121 *** Got signal Terminated *** 2024-04-13T03:52:45.604 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3[50416]: 2024-04-13T03:52:45.197+0000 7f92dda00700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:52:46.354 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:46 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3[50416]: 2024-04-13T03:52:46.103+0000 7f92d9200700 -1 osd.3 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.420638+0000 front 2024-04-13T03:52:20.420105+0000 (oldest deadline 2024-04-13T03:52:45.720269+0000) 2024-04-13T03:52:46.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:45 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:45.981+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:46.372 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:46 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:46.029+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:47.354 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:47 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3[50416]: 2024-04-13T03:52:47.056+0000 7f92d9200700 -1 osd.3 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.420638+0000 front 2024-04-13T03:52:20.420105+0000 (oldest deadline 2024-04-13T03:52:45.720269+0000) 2024-04-13T03:52:47.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:46 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:46.973+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:47.372 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:46 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:46.982+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:47.570 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:47 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:47.355+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:48.354 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:48 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3[50416]: 2024-04-13T03:52:48.086+0000 7f92d9200700 -1 osd.3 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.420638+0000 front 2024-04-13T03:52:20.420105+0000 (oldest deadline 2024-04-13T03:52:45.720269+0000) 2024-04-13T03:52:48.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:47 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:47.938+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:48.372 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:47 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:47.949+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:48.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:48 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:48.327+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:49.255 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:49 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3[50416]: 2024-04-13T03:52:49.131+0000 7f92d9200700 -1 osd.3 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.420638+0000 front 2024-04-13T03:52:20.420105+0000 (oldest deadline 2024-04-13T03:52:45.720269+0000) 2024-04-13T03:52:49.256 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:48 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:48.960+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:49.372 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:48 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:48.932+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:49.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:49 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:49.365+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:50.354 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:50 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3[50416]: 2024-04-13T03:52:50.156+0000 7f92d9200700 -1 osd.3 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.420638+0000 front 2024-04-13T03:52:20.420105+0000 (oldest deadline 2024-04-13T03:52:45.720269+0000) 2024-04-13T03:52:50.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:50 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:50.000+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:50.372 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:49 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:49.914+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:50.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:50 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:50.337+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:51.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:51 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:51.021+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:51.104 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:50 smithi016 podman[91741]: 2024-04-13 03:52:50.849081483 +0000 UTC m=+5.807739825 container stop 80ad07a7b56e8fd36f89e30f3488bb08e9d191483dad5d9948f1cafcf0fea61a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:52:51.104 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:50 smithi016 podman[91741]: 2024-04-13 03:52:50.875087817 +0000 UTC m=+5.833746159 container died 80ad07a7b56e8fd36f89e30f3488bb08e9d191483dad5d9948f1cafcf0fea61a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-13T03:52:51.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:50 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:50.888+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:51.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:50 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:50.781+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:52:51.600 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:51 smithi016 podman[91741]: 2024-04-13 03:52:51.348918104 +0000 UTC m=+6.307576487 container cleanup 80ad07a7b56e8fd36f89e30f3488bb08e9d191483dad5d9948f1cafcf0fea61a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-13T03:52:51.601 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:51 smithi016 bash[91741]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3 2024-04-13T03:52:51.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:51 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:51.352+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:51.853 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:51 smithi016 podman[91760]: 2024-04-13 03:52:51.600199284 +0000 UTC m=+0.725542360 container remove 80ad07a7b56e8fd36f89e30f3488bb08e9d191483dad5d9948f1cafcf0fea61a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-04-13T03:52:52.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:51 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:51.918+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:52.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:51 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:51.813+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:52:52.353 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:51 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:51.982+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:52.664 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:52 smithi016 podman[91843]: 2024-04-13 03:52:52.318324817 +0000 UTC m=+0.037160769 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:52:52.664 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:52 smithi016 podman[91843]: 2024-04-13 03:52:52.492887652 +0000 UTC m=+0.211723641 container create 1d718e0f8bb18a8d5ac47f2e105bce28869eb079b355b0ddb3143b8245181e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-13T03:52:52.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:52 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:52.401+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:53.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:53 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:53.029+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:53.104 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:52 smithi016 podman[91843]: 2024-04-13 03:52:52.809385733 +0000 UTC m=+0.528221765 container init 1d718e0f8bb18a8d5ac47f2e105bce28869eb079b355b0ddb3143b8245181e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-13T03:52:53.105 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:52 smithi016 podman[91843]: 2024-04-13 03:52:52.823460569 +0000 UTC m=+0.542296564 container start 1d718e0f8bb18a8d5ac47f2e105bce28869eb079b355b0ddb3143b8245181e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-13T03:52:53.105 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:52 smithi016 podman[91843]: 2024-04-13 03:52:52.929112177 +0000 UTC m=+0.647948220 container attach 1d718e0f8bb18a8d5ac47f2e105bce28869eb079b355b0ddb3143b8245181e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-13T03:52:53.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:52 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:52.902+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:53.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:52 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:52.850+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:52:53.604 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 13 03:52:53 smithi016 podman[91868]: 2024-04-13 03:52:53.352429012 +0000 UTC m=+0.032638227 container died 1d718e0f8bb18a8d5ac47f2e105bce28869eb079b355b0ddb3143b8245181e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:52:53.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:53 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:53.374+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:54.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:53.868+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:54.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:53.882+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:52:54.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.3.service' 2024-04-13T03:52:54.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:52:54.256 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-13T03:52:54.256 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-13T03:52:54.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.4 2024-04-13T03:52:54.353 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:54 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:54.024+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:54.564 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:54 smithi119 systemd[1]: Stopping Ceph osd.4 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:52:54.564 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:54 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:54.373+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:54.820 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:54 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:52:54.563+0000 7f2821b02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:52:54.821 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:54 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:52:54.563+0000 7f2821b02700 -1 osd.4 121 *** Got signal Terminated *** 2024-04-13T03:52:54.821 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:54 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:52:54.563+0000 7f2821b02700 -1 osd.4 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:52:55.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:54.825+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:55.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:54.860+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:52:55.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:54.860+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:52:55.353 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:54 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:54.976+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:55.570 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:55 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:52:55.181+0000 7f281e0f6700 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.014636+0000 front 2024-04-13T03:52:32.014666+0000 (oldest deadline 2024-04-13T03:52:54.914799+0000) 2024-04-13T03:52:55.571 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:55 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:55.402+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:55.571 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:55 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:55.402+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:52:56.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:55.817+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:52:56.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:55.817+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:52:56.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:55.816+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:56.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:55.816+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:52:56.353 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:55 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:55.931+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:56.432 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:56 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:52:56.168+0000 7f281e0f6700 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.014636+0000 front 2024-04-13T03:52:32.014666+0000 (oldest deadline 2024-04-13T03:52:54.914799+0000) 2024-04-13T03:52:56.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:52:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:52:56.356+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:52:56.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:56 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:56.432+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:56.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:56 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:56.432+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:52:57.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:56.798+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:52:57.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:56.798+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:52:57.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:56.825+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:57.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:56.825+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:52:57.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:56 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:56.885+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:57.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:56 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:56.885+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:52:57.463 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:57 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:52:57.121+0000 7f281e0f6700 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.014636+0000 front 2024-04-13T03:52:32.014666+0000 (oldest deadline 2024-04-13T03:52:54.914799+0000) 2024-04-13T03:52:57.464 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:52:57 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:52:57.160+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:52:57.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:52:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:52:57.365+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:52:57.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:57 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:57.464+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:57.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:57 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:57.464+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:52:58.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:57.806+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:58.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:57.806+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:52:58.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:57.774+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:52:58.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:57.774+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:52:58.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:57 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:57.875+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:58.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:57 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:57.875+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:52:58.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:52:58 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:52:58.014+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:52:58.485 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:52:58 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:52:58.165+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:52:58.485 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:58 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:52:58.154+0000 7f281e0f6700 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.014636+0000 front 2024-04-13T03:52:32.014666+0000 (oldest deadline 2024-04-13T03:52:54.914799+0000) 2024-04-13T03:52:58.764 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:52:58 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:52:58.390+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:52:58.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:58 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:58.485+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:58.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:58 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:58.485+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:52:59.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:58 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:58.818+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:52:59.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:58 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:58.818+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:52:59.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:58 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:58.764+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:52:59.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:58 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:58.764+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:52:59.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:52:58 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:52:58.971+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:52:59.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:58 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:58.888+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:52:59.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:58 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:58.888+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:52:59.471 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:59 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4[46929]: 2024-04-13T03:52:59.179+0000 7f281e0f6700 -1 osd.4 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.014636+0000 front 2024-04-13T03:52:32.014666+0000 (oldest deadline 2024-04-13T03:52:54.914799+0000) 2024-04-13T03:52:59.472 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:52:59 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:52:59.160+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:52:59.787 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:52:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:52:59.375+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:52:59.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:59 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:59.472+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:52:59.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:52:59 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:52:59.472+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:00.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:59.812+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:53:00.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:52:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:52:59.812+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:53:00.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:59.786+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:00.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:52:59 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:52:59.786+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:00.207 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:52:59 smithi119 podman[74050]: 2024-04-13 03:52:59.947449511 +0000 UTC m=+5.497054707 container died 684bddd74810f658aa7afbebc6d897170f0c944fb44fcb3f9d0f443d2b7c5a32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-13T03:53:00.207 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:00 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:00.192+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:00.353 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:52:59 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:52:59.997+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:00.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:59 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:59.925+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:00.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:52:59 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:52:59.925+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:00.477 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:53:00 smithi119 podman[74050]: 2024-04-13 03:53:00.289086488 +0000 UTC m=+5.838691637 container cleanup 684bddd74810f658aa7afbebc6d897170f0c944fb44fcb3f9d0f443d2b7c5a32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409) 2024-04-13T03:53:00.477 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:53:00 smithi119 bash[74050]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4 2024-04-13T03:53:00.765 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:00.411+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:00.820 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:53:00 smithi119 podman[74070]: 2024-04-13 03:53:00.532212779 +0000 UTC m=+0.580192541 container remove 684bddd74810f658aa7afbebc6d897170f0c944fb44fcb3f9d0f443d2b7c5a32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4, org.label-schema.build-date=20240409, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2) 2024-04-13T03:53:00.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:00 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:00.477+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:00.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:00 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:00.477+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:01.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:00.765+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:01.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:00.765+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:01.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:00.835+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:53:01.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:00 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:00.835+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:53:01.194 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:53:00 smithi119 podman[74161]: 2024-04-13 03:53:00.843747678 +0000 UTC m=+0.017977944 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:53:01.194 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:53:00 smithi119 podman[74161]: 2024-04-13 03:53:00.941772093 +0000 UTC m=+0.116002372 container create eaaef1ce311d79d6de8b2bd1553f055bbcd6d13bf455f4107411b1a3bda48ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-04-13T03:53:01.194 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:53:01 smithi119 podman[74161]: 2024-04-13 03:53:01.177568271 +0000 UTC m=+0.351798530 container init eaaef1ce311d79d6de8b2bd1553f055bbcd6d13bf455f4107411b1a3bda48ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD) 2024-04-13T03:53:01.195 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:53:01 smithi119 podman[74161]: 2024-04-13 03:53:01.180620453 +0000 UTC m=+0.354850713 container start eaaef1ce311d79d6de8b2bd1553f055bbcd6d13bf455f4107411b1a3bda48ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-13T03:53:01.353 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:00.972+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:01.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:00.932+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:01.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:00 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:00.932+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:01.469 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:01 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:01.194+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:01.470 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:53:01 smithi119 podman[74161]: 2024-04-13 03:53:01.269342868 +0000 UTC m=+0.443573142 container attach eaaef1ce311d79d6de8b2bd1553f055bbcd6d13bf455f4107411b1a3bda48ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:53:01.764 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 13 03:53:01 smithi119 podman[74161]: 2024-04-13 03:53:01.470677489 +0000 UTC m=+0.644907775 container died eaaef1ce311d79d6de8b2bd1553f055bbcd6d13bf455f4107411b1a3bda48ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, ceph=True, GIT_CLEAN=True) 2024-04-13T03:53:01.764 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:01 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:01.504+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:01.764 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:01 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:01.504+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:01.812 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:01 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:01.387+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:02.050 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.4.service' 2024-04-13T03:53:02.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:53:02.091 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-13T03:53:02.092 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-13T03:53:02.092 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.5 2024-04-13T03:53:02.120 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:01 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:01.811+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:02.121 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:01 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:01.811+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:02.121 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:01 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:01.839+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:53:02.121 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:01 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:01.839+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:53:02.320 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:02 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:02.235+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:02.353 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:01 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:01.932+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:02.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:01 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:01.889+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:02.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:01 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:01.889+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:02.372 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:02 smithi188 systemd[1]: Stopping Ceph osd.5 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:53:02.790 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:02.375+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:02.791 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:02.407+0000 7f94f6d4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:53:02.791 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:02.407+0000 7f94f6d4e700 -1 osd.5 121 *** Got signal Terminated *** 2024-04-13T03:53:02.791 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:02.407+0000 7f94f6d4e700 -1 osd.5 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:53:02.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:02 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:02.544+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:02.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:02 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:02.544+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:03.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:02 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:02.952+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:03.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:02 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:02.952+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:03.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:02 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:02.852+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:03.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:02 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:02.852+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:03.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:02 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:02.852+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:03.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:02.800+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:53:03.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:02.800+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:53:03.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:02.790+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:03.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:02 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:02.790+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:03.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:03 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:03.279+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:03.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:03 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:03.279+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:03.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:03 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:03.362+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:03.820 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:03 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:03.568+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:03.821 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:03 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:03.568+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:04.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:03 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:03.918+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:04.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:03 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:03.918+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:04.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:03 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:03.827+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:04.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:03 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:03.827+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:04.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:03 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:03.827+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:04.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:03 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:03.795+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:53:04.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:03 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:03.795+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:53:04.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:03 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:03.817+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:04.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:03 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:03.817+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:04.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:04.295+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:04.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:04.295+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:04.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:04.318+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:05.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:04.580+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:05.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:04 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:04.580+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:05.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:04.763+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:53:05.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:04.763+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:53:05.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:04.789+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:05.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:04 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:04.789+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:05.341 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:05 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:05.268+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:05.343 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:05 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:05.268+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:05.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:04.916+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:05.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:04.916+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:05.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:04.863+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:05.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:04.863+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:05.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:04 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:04.863+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:05.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:05 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:05.315+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:06.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:05 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:05.615+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:06.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:05 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:05.615+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:06.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:05 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:05.757+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:06.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:05 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:05.757+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:06.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:05 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:05.780+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:53:06.123 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:05 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:05.780+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:53:06.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:05 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:05.872+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:06.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:05 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:05.872+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:06.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:05 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:05.912+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:06.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:05 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:05.912+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:06.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:05 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:05.912+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:06.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:06 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:06.295+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:06.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:06 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:06.295+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:06.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:06 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:06.361+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:07.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:06 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:06.592+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:07.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:06 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:06.592+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:07.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:06 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:06.592+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:07.103 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:06.841+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:07.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:06.841+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:07.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:06.938+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:07.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:06.938+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:07.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:06.938+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:07.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:06 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:06.771+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:21.436470+0000 front 2024-04-13T03:52:21.436610+0000 (oldest deadline 2024-04-13T03:52:45.536356+0000) 2024-04-13T03:53:07.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:06 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5[48164]: 2024-04-13T03:53:06.771+0000 7f94f3342700 -1 osd.5 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.138291+0000 front 2024-04-13T03:52:33.138196+0000 (oldest deadline 2024-04-13T03:52:54.838232+0000) 2024-04-13T03:53:07.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:06 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:06.780+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:07.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:06 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:06.780+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:07.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:06 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:06.780+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:07.394 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:07 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:07.374+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:07.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:07.299+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:07.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:07.299+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:08.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:07.620+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:08.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:07.620+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:08.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:07 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:07.620+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:08.122 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:07 smithi188 podman[76372]: 2024-04-13 03:53:07.899583984 +0000 UTC m=+5.618552895 container died 49b7a8e944836c4fa7d2c1adc3ae1496589cb2f9445c408607cd024a3fccb025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:53:08.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:07 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:07.767+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:08.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:07 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:07.767+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:08.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:07 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:07.767+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:08.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:07 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:07.865+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:08.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:07 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:07.865+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:08.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:07 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:07.928+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:08.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:07 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:07.928+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:08.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:07 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:07.928+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:08.402 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:08 smithi188 podman[76372]: 2024-04-13 03:53:08.207490223 +0000 UTC m=+5.926459124 container cleanup 49b7a8e944836c4fa7d2c1adc3ae1496589cb2f9445c408607cd024a3fccb025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:53:08.402 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:08 smithi188 bash[76372]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5 2024-04-13T03:53:08.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:08 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:08.305+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:08.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:08 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:08.305+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:08.765 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:08 smithi188 podman[76400]: 2024-04-13 03:53:08.467885324 +0000 UTC m=+0.568485415 container remove 49b7a8e944836c4fa7d2c1adc3ae1496589cb2f9445c408607cd024a3fccb025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:53:08.766 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:08 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:08.402+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:08.766 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:08 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:08.402+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:09.062 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:08 smithi188 podman[76493]: 2024-04-13 03:53:08.802143037 +0000 UTC m=+0.104177401 container create 41132e06b85663538c67c71079d9cf344137d0c8df775f1e9fe442646ed87457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-13T03:53:09.063 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:08 smithi188 podman[76493]: 2024-04-13 03:53:08.715853551 +0000 UTC m=+0.017887924 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:53:09.063 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:09 smithi188 podman[76493]: 2024-04-13 03:53:09.062997699 +0000 UTC m=+0.365032071 container init 41132e06b85663538c67c71079d9cf344137d0c8df775f1e9fe442646ed87457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:53:09.063 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:08 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:08.764+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:09.064 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:08 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:08.764+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:09.064 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:08 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:08.764+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:09.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:08 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:08.650+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:09.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:08 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:08.650+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:09.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:08 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:08.650+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:09.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:08 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:08.836+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:09.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:08 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:08.836+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:09.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:08 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:08.934+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:09.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:08 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:08.934+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:09.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:08 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:08.934+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:09.354 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:09 smithi188 podman[76493]: 2024-04-13 03:53:09.066265093 +0000 UTC m=+0.368299464 container start 41132e06b85663538c67c71079d9cf344137d0c8df775f1e9fe442646ed87457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-13T03:53:09.355 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:09 smithi188 podman[76493]: 2024-04-13 03:53:09.162956616 +0000 UTC m=+0.464991000 container attach 41132e06b85663538c67c71079d9cf344137d0c8df775f1e9fe442646ed87457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-13T03:53:09.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:09 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:09.260+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:09.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:09 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:09.260+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:09.622 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 13 03:53:09 smithi188 podman[76493]: 2024-04-13 03:53:09.355419707 +0000 UTC m=+0.657454096 container died 41132e06b85663538c67c71079d9cf344137d0c8df775f1e9fe442646ed87457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-13T03:53:09.623 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:09 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:09.365+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:09.623 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:09 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:09.365+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:09.916 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:09 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:09.727+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:09.916 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:09 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:09.727+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:09.916 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:09 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:09.727+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:09.939 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.5.service' 2024-04-13T03:53:10.023 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:53:10.024 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-13T03:53:10.024 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-13T03:53:10.024 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.6 2024-04-13T03:53:10.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:09 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:09.619+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:10.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:09 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:09.619+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:10.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:09 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:09.619+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:10.103 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:09 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:09.853+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:10.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:09 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:09.853+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:10.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:09 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:09.962+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:10.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:09 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:09.962+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:10.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:09 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:09.962+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:10.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:10 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:10.266+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:10.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:10 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:10.266+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:10.604 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:10 smithi016 systemd[1]: Stopping Ceph osd.6 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:53:10.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:10 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:10.324+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:10.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:10 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:10.324+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:10.966 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:10.634+0000 7f1bba000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:53:10.967 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:10.634+0000 7f1bba000700 -1 osd.6 121 *** Got signal Terminated *** 2024-04-13T03:53:10.967 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:10.634+0000 7f1bba000700 -1 osd.6 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:53:10.967 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:10.815+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:10.967 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:10.815+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:11.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:10 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:10.580+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:11.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:10 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:10.580+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:11.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:10 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:10.580+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:11.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:10 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:10.762+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:11.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:10 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:10.762+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:11.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:10 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:10.762+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:11.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:10.965+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:11.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:10.965+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:11.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:10.965+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:11.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:11 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:11.220+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:11.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:11 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:11.220+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:11.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:11 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:11.318+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:11.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:11 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:11.318+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:12.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:11 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:11.606+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:12.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:11 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:11.606+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:12.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:11 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:11.606+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:12.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:11 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:11.606+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:12.103 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:11 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:11.799+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:12.103 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:11 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:11.799+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:12.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:11 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:11.920+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:12.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:11 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:11.920+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:12.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:11 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:11.920+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:12.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:11 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:11.810+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:12.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:11 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:11.810+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:12.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:11 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:11.810+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:12.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:12 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:12.182+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:12.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:12 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:12.182+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:12.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:12 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:12.351+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:12.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:12 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:12.351+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:12.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:12 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:12.351+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:13.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:12 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:12.600+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:13.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:12 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:12.600+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:13.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:12 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:12.600+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:13.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:12 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:12.600+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:13.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:12 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:12.777+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:13.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:12 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:12.777+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:13.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:12 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:12.777+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.030517+0000 front 2024-04-13T03:52:48.030638+0000 (oldest deadline 2024-04-13T03:53:12.130546+0000) 2024-04-13T03:53:13.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:12 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:12.889+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:13.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:12 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:12.889+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:13.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:12 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:12.889+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:13.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:12 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:12.889+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:13.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:12 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:12.856+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:13.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:12 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:12.856+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:13.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:12 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:12.856+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:13.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:13 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:13.207+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:13.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:13 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:13.207+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:13.849 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:13 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:13.372+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:13.850 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:13 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:13.372+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:13.850 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:13 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:13.372+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:14.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:13 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:13.576+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:14.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:13 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:13.576+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:14.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:13 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:13.576+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:14.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:13 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:13.576+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:14.103 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:13 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:13.744+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:14.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:13 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:13.744+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:14.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:13 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:13.744+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.030517+0000 front 2024-04-13T03:52:48.030638+0000 (oldest deadline 2024-04-13T03:53:12.130546+0000) 2024-04-13T03:53:14.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:13 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:13.929+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:14.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:13 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:13.929+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:14.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:13 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:13.929+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:14.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:13 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:13.929+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:14.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:13 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:13.849+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:14.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:13 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:13.849+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:14.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:13 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:13.849+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:14.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:14 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:14.209+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:14.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:14 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:14.209+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:14.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:14 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:14.209+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:14.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:14 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:14.325+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:14.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:14 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:14.325+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:14.623 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:14 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:14.325+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:15.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:14 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:14.625+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:15.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:14 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:14.625+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:15.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:14 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:14.625+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:15.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:14 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:14.625+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:15.103 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:14.757+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:32.922213+0000 front 2024-04-13T03:52:32.922121+0000 (oldest deadline 2024-04-13T03:52:57.622268+0000) 2024-04-13T03:53:15.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:14.757+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:39.926430+0000 front 2024-04-13T03:52:39.926526+0000 (oldest deadline 2024-04-13T03:53:02.826673+0000) 2024-04-13T03:53:15.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6[57138]: 2024-04-13T03:53:14.757+0000 7f1bb5800700 -1 osd.6 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.030517+0000 front 2024-04-13T03:52:48.030638+0000 (oldest deadline 2024-04-13T03:53:12.130546+0000) 2024-04-13T03:53:15.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:14.965+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:15.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:14.965+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:15.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:14.965+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:15.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:14.965+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:15.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:14 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:14.870+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:15.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:14 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:14.870+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:15.123 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:14 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:14.870+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:15.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:15 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:15.203+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:15.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:15 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:15.203+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:15.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:15 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:15.203+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:15.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:15 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:15.313+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:15.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:15 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:15.313+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:15.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:15 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:15.313+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:16.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:15 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:15.644+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:16.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:15 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:15.644+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:16.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:15 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:15.644+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:16.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:15 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:15.644+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:16.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:15 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:15.828+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:16.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:15 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:15.828+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:16.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:15 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:15.828+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:16.303 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:15 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:15.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:16.303 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:15 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:15.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:16.303 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:15 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:15.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:16.303 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:15 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:15.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:16.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:16 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:16.243+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:16.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:16 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:16.243+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:16.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:16 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:16.243+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:16.603 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:16 smithi016 podman[91948]: 2024-04-13 03:53:16.303037195 +0000 UTC m=+5.809488252 container died 4fe2a2c64ea6fab2c60b49249f78c0a4ec567760fa712fb10812af7971b77bf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-13T03:53:16.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:16 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:16.266+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:16.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:16 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:16.266+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:16.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:16 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:16.266+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:17.011 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:16 smithi016 podman[91948]: 2024-04-13 03:53:16.751689997 +0000 UTC m=+6.258141091 container cleanup 4fe2a2c64ea6fab2c60b49249f78c0a4ec567760fa712fb10812af7971b77bf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409) 2024-04-13T03:53:17.011 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:16 smithi016 bash[91948]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6 2024-04-13T03:53:17.011 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:16 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:16.873+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:17.012 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:16 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:16.873+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:17.012 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:16 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:16.873+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:17.012 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:16 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:16.873+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:17.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:16 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:16.639+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:17.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:16 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:16.639+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:17.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:16 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:16.639+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:17.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:16 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:16.639+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:17.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:16 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:16.856+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:17.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:16 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:16.856+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:17.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:16 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:16.856+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:17.353 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:17 smithi016 podman[91969]: 2024-04-13 03:53:17.010933538 +0000 UTC m=+0.693395361 container remove 4fe2a2c64ea6fab2c60b49249f78c0a4ec567760fa712fb10812af7971b77bf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-13T03:53:17.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:17 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:17.202+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:17.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:17 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:17.202+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:17.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:17 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:17.202+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:17.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:17 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:17.256+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:17.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:17 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:17.256+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:17.623 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:17 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:17.256+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:18.058 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:17 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:17.879+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:18.059 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:17 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:17.879+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:18.059 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:17 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:17.879+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:18.059 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:17 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:17.879+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:18.059 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:17 smithi016 podman[92061]: 2024-04-13 03:53:17.711295998 +0000 UTC m=+0.032226259 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:53:18.059 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:17 smithi016 podman[92061]: 2024-04-13 03:53:17.895446331 +0000 UTC m=+0.216376629 container create 66e8bff81c3d822af7e03592d83d83891d604226af5a9f238eb00c3ce014196a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True) 2024-04-13T03:53:18.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:17 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:17.621+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:18.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:17 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:17.621+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:18.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:17 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:17.621+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:18.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:17 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:17.621+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:18.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:17 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:17.866+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:18.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:17 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:17.866+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:18.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:17 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:17.866+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:18.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:18 smithi016 podman[92061]: 2024-04-13 03:53:18.215799081 +0000 UTC m=+0.536729386 container init 66e8bff81c3d822af7e03592d83d83891d604226af5a9f238eb00c3ce014196a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-13T03:53:18.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:18 smithi016 podman[92061]: 2024-04-13 03:53:18.2276732 +0000 UTC m=+0.548603504 container start 66e8bff81c3d822af7e03592d83d83891d604226af5a9f238eb00c3ce014196a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-13T03:53:18.354 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:18 smithi016 podman[92061]: 2024-04-13 03:53:18.306230589 +0000 UTC m=+0.627160907 container attach 66e8bff81c3d822af7e03592d83d83891d604226af5a9f238eb00c3ce014196a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-13T03:53:18.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:18 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:18.231+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:18.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:18 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:18.231+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:18.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:18 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:18.231+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:18.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:18 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:18.272+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:18.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:18 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:18.272+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:18.623 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:18 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:18.272+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:19.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:18 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:18.628+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:19.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:18 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:18.628+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:19.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:18 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:18.628+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:19.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:18 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:18.628+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:19.104 INFO:journalctl@ceph.osd.6.smithi016.stdout:Apr 13 03:53:18 smithi016 podman[92061]: 2024-04-13 03:53:18.692314919 +0000 UTC m=+1.013245223 container died 66e8bff81c3d822af7e03592d83d83891d604226af5a9f238eb00c3ce014196a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2) 2024-04-13T03:53:19.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:18 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:18.883+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:19.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:18 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:18.883+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:19.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:18 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:18.883+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:19.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:18 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:18.883+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:19.260 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:18 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:18.911+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:19.260 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:18 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:18.911+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:19.260 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:18 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:18.911+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:19.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.6.service' 2024-04-13T03:53:19.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:19.219+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:19.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:19.219+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:19.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:19.219+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:19.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:19 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:19.260+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:19.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:19 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:19.260+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:19.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:19 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:19.260+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:19.664 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:53:19.664 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-13T03:53:19.664 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-13T03:53:19.665 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.7 2024-04-13T03:53:19.975 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:19 smithi119 systemd[1]: Stopping Ceph osd.7 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:53:19.976 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:19.628+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:19.976 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:19.628+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:19.976 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:19.628+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:19.976 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:19.628+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:20.305 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:19 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:19.950+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:20.305 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:19 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:19.950+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:20.305 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:19 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:19.950+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:20.320 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:19.975+0000 7f3b26ce0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:53:20.321 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:19.975+0000 7f3b26ce0700 -1 osd.7 121 *** Got signal Terminated *** 2024-04-13T03:53:20.321 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:19 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:19.975+0000 7f3b26ce0700 -1 osd.7 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:53:20.321 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:20 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:20.223+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:20.321 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:20 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:20.223+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:20.321 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:20 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:20.223+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:20.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:19 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:19.877+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:20.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:19 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:19.877+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:20.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:19 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:19.877+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:20.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:19 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:19.877+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:20.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:20 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:20.304+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:20.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:20 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:20.304+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:20.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:20 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:20.304+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:21.065 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:20 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:20.877+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:21.065 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:20 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:20.877+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:21.065 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:20 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:20.877+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:21.065 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:20 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:20.877+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:21.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:20 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:20.607+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:21.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:20 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:20.607+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:21.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:20 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:20.607+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:21.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:20 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:20.607+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:21.305 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:20 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:20.943+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:21.310 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:20 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:20.943+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:21.310 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:20 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:20.943+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:21.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:21 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:21.173+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:21.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:21 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:21.173+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:21.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:21 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:21.173+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:21.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:21 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:21.305+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:21.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:21 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:21.305+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:21.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:21 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:21.305+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:22.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:21 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:21.607+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:22.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:21 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:21.607+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:22.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:21 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:21.607+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:22.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:21 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:21.607+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:22.329 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:21 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:21.922+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:22.329 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:21 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:21.922+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:22.329 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:21 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:21.922+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:22.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:21 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:21.889+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:22.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:21 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:21.889+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:22.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:21 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:21.890+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:22.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:21 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:21.890+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:22.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:22 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:22.161+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:22.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:22 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:22.161+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:22.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:22 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:22.161+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:22.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:22 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:22.161+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.944036+0000 front 2024-04-13T03:52:56.944073+0000 (oldest deadline 2024-04-13T03:53:21.544647+0000) 2024-04-13T03:53:22.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:22 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:22.329+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:22.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:22 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:22.329+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:22.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:22 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:22.329+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:23.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:22 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:22.611+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:23.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:22 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:22.611+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:23.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:22 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:22.611+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:23.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:22 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:22.611+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:23.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:22 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:22.611+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:23.334 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:22 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:22.948+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:23.334 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:22 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:22.948+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:23.334 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:22 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:22.948+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:23.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:22 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:22.878+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:23.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:22 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:22.878+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:23.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:22 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:22.878+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:23.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:22 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:22.878+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:23.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:23 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:23.148+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:23.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:23 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:23.148+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:23.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:23 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:23.148+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:23.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:23 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:23.148+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.944036+0000 front 2024-04-13T03:52:56.944073+0000 (oldest deadline 2024-04-13T03:53:21.544647+0000) 2024-04-13T03:53:23.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:23 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:23.334+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:23.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:23 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:23.334+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:23.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:23 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:23.334+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:23.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:23 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:23.334+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:24.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:23 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:23.629+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:24.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:23 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:23.629+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:24.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:23 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:23.629+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:24.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:23 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:23.629+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:24.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:23 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:23.629+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:24.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:23 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:23.869+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:24.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:23 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:23.869+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:24.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:23 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:23.869+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:24.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:23 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:23.869+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:24.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:23 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:23.869+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:24.366 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:23 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:23.965+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:24.366 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:23 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:23.965+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:24.366 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:23 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:23.965+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:24.570 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:24 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:24.166+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:33.238099+0000 front 2024-04-13T03:52:33.238129+0000 (oldest deadline 2024-04-13T03:52:56.738387+0000) 2024-04-13T03:53:24.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:24 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:24.166+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:40.140556+0000 front 2024-04-13T03:52:40.140402+0000 (oldest deadline 2024-04-13T03:53:03.040385+0000) 2024-04-13T03:53:24.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:24 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:24.166+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:45.342046+0000 front 2024-04-13T03:52:45.341999+0000 (oldest deadline 2024-04-13T03:53:13.542592+0000) 2024-04-13T03:53:24.571 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:24 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7[53510]: 2024-04-13T03:53:24.166+0000 7f3b1f2cc700 -1 osd.7 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.944036+0000 front 2024-04-13T03:52:56.944073+0000 (oldest deadline 2024-04-13T03:53:21.544647+0000) 2024-04-13T03:53:24.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:24 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:24.366+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:24.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:24 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:24.366+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:24.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:24 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:24.366+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:24.623 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:24 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:24.366+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:25.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:24 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:24.669+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:25.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:24 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:24.669+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:25.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:24 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:24.669+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:25.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:24 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:24.669+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:25.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:24 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:24.669+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:25.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:24 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:24.893+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:25.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:24 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:24.893+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:25.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:24 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:24.893+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:25.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:24 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:24.893+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:25.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:24 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:24.893+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:25.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:24.999+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:25.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:24.999+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:25.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:24.999+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:25.684 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:25 smithi119 podman[74280]: 2024-04-13 03:53:25.419603889 +0000 UTC m=+5.561833414 container died 09d960de05e544e27bafd65912715b3b86d257e0fad993d9461f90a8e8dee36e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-13T03:53:25.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:25.388+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:25.873 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:25.388+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:25.873 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:25.388+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:25.873 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:25.388+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:26.070 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:25 smithi119 podman[74280]: 2024-04-13 03:53:25.775417609 +0000 UTC m=+5.917647141 container cleanup 09d960de05e544e27bafd65912715b3b86d257e0fad993d9461f90a8e8dee36e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-13T03:53:26.071 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:25 smithi119 bash[74280]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7 2024-04-13T03:53:26.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:25 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:25.686+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:26.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:25 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:25.686+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:26.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:25 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:25.686+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:26.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:25 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:25.686+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:26.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:25 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:25.686+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:26.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:25 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:25.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:26.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:25 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:25.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:26.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:25 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:25.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:26.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:25 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:25.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:26.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:25 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:25.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:26.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:25.980+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:26.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:25.980+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:26.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:25.980+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:26.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:25 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:25.980+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:26.437 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:26 smithi119 podman[74298]: 2024-04-13 03:53:26.101693102 +0000 UTC m=+0.684552713 container remove 09d960de05e544e27bafd65912715b3b86d257e0fad993d9461f90a8e8dee36e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-13T03:53:26.713 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:26 smithi119 podman[74390]: 2024-04-13 03:53:26.340358546 +0000 UTC m=+0.017660850 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:53:26.713 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:26 smithi119 podman[74390]: 2024-04-13 03:53:26.469250314 +0000 UTC m=+0.146552611 container create 9edb6ec64d59c9f24cfbccb806e13ae990757c68ff4016454da2510b9e900c6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-13T03:53:26.713 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:26 smithi119 podman[74390]: 2024-04-13 03:53:26.664252531 +0000 UTC m=+0.341554831 container init 9edb6ec64d59c9f24cfbccb806e13ae990757c68ff4016454da2510b9e900c6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-13T03:53:26.714 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:26 smithi119 podman[74390]: 2024-04-13 03:53:26.667642193 +0000 UTC m=+0.344944495 container start 9edb6ec64d59c9f24cfbccb806e13ae990757c68ff4016454da2510b9e900c6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-04-13T03:53:26.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:26 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:26.390+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:26.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:26 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:26.390+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:26.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:26 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:26.390+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:26.873 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:26 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:26.390+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:27.071 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:26 smithi119 podman[74390]: 2024-04-13 03:53:26.772181926 +0000 UTC m=+0.449484230 container attach 9edb6ec64d59c9f24cfbccb806e13ae990757c68ff4016454da2510b9e900c6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:53:27.071 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 13 03:53:26 smithi119 podman[74390]: 2024-04-13 03:53:26.957783277 +0000 UTC m=+0.635085581 container died 9edb6ec64d59c9f24cfbccb806e13ae990757c68ff4016454da2510b9e900c6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-13T03:53:27.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:26 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:26.712+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:27.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:26 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:26.712+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:27.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:26 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:26.712+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:27.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:26 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:26.712+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:27.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:26 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:26.712+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:27.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:26 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:26.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:27.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:26 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:26.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:27.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:26 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:26.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:27.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:26 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:26.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:27.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:26 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:26.943+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:27.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:26 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:26.970+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:27.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:26 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:26.970+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:27.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:26 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:26.970+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:27.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:26 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:26.970+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:27.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.7.service' 2024-04-13T03:53:27.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:53:27.622 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-13T03:53:27.622 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-13T03:53:27.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.8 2024-04-13T03:53:27.653 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:27.376+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:27.653 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:27.376+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:27.653 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:27.376+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:27.653 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:27.376+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:27.822 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:27 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:27.756+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:27.823 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:27 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:27.756+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:27.823 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:27 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:27.756+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:27.824 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:27 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:27.756+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:27.824 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:27 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:27.756+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:27.928 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:27 smithi188 systemd[1]: Stopping Ceph osd.8 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:53:28.341 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:27.928+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:28.342 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:27.928+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:28.342 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:27.928+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:28.342 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:27.928+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:28.342 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:27.927+0000 7f53c3b4f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:53:28.342 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:27.927+0000 7f53c3b4f700 -1 osd.8 121 *** Got signal Terminated *** 2024-04-13T03:53:28.342 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:27 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:27.927+0000 7f53c3b4f700 -1 osd.8 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:53:28.343 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:28 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:28.341+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:28.343 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:28 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:28.341+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:28.343 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:28 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:28.341+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:28.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:27 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:27.901+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:28.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:27 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:27.901+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:28.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:27 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:27.901+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:28.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:27 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:27.901+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:28.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:27 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:27.901+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:28.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:28 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:28.341+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:29.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:28 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:28.725+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:29.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:28 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:28.725+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:29.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:28 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:28.725+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:29.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:28 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:28.725+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:29.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:28 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:28.725+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:29.340 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:28 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:28.943+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:29.340 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:28 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:28.943+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:29.340 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:28 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:28.943+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:29.341 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:28 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:28.943+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:29.341 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:28 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:28.943+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:29.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:28 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:28.862+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:29.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:28 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:28.862+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:29.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:28 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:28.862+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:29.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:28 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:28.862+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:29.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:28 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:28.862+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:29.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:29 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:29.339+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:29.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:29 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:29.339+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:29.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:29 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:29.339+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:29.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:29 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:29.339+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:30.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:29 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:29.737+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:30.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:29 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:29.737+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:30.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:29 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:29.737+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:30.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:29 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:29.737+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:30.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:29 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:29.737+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:30.353 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:29 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:29.867+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:30.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:29 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:29.867+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:30.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:29 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:29.867+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:30.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:29 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:29.867+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:30.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:29 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:29.867+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:30.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:29 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:29.867+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:30.371 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:29 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:29.983+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:30.371 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:29 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:29.983+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:30.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:29 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:29.983+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:30.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:29 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:29.983+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:30.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:29 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:29.983+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:30.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:30 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:30.371+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:30.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:30 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:30.371+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:30.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:30 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:30.371+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:30.623 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:30 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:30.371+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:31.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:30 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:30.711+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:31.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:30 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:30.711+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:31.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:30 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:30.711+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:31.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:30 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:30.711+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:31.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:30 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:30.711+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:31.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:30 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:30.897+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:31.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:30 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:30.897+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:31.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:30 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:30.897+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:31.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:30 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:30.897+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:31.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:30 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:30.897+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:31.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:30 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:30.897+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:31.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:30 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:30.964+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:31.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:30 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:30.964+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:31.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:30 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:30.964+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:31.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:30 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:30.964+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:31.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:30 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:30.964+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:31.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:31 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:31.397+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:31.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:31 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:31.397+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:31.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:31 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:31.397+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:31.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:31 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:31.397+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:32.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:31 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:31.752+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:32.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:31 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:31.752+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:32.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:31 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:31.752+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:32.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:31 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:31.752+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:32.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:31 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:31.752+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:32.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:31 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:31.752+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:32.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:31.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:32.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:31.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:32.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:31.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:32.104 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:31.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:32.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:31.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:32.105 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:31 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:31.916+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:32.349 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:31 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:31.965+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:32.350 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:31 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:31.965+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:32.350 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:31 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:31.965+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:32.350 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:31 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:31.965+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:32.350 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:31 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:31.965+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:32.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:32.350+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.924551+0000 front 2024-04-13T03:52:31.924707+0000 (oldest deadline 2024-04-13T03:52:56.024671+0000) 2024-04-13T03:53:32.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:32.350+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.726820+0000 front 2024-04-13T03:52:41.726793+0000 (oldest deadline 2024-04-13T03:53:07.627020+0000) 2024-04-13T03:53:32.622 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:32.350+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:48.128469+0000 front 2024-04-13T03:52:48.128642+0000 (oldest deadline 2024-04-13T03:53:11.628279+0000) 2024-04-13T03:53:32.623 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:32.350+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.430308+0000 front 2024-04-13T03:52:57.430242+0000 (oldest deadline 2024-04-13T03:53:23.330254+0000) 2024-04-13T03:53:32.623 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8[55174]: 2024-04-13T03:53:32.350+0000 7f53bc13b700 -1 osd.8 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:06.831412+0000 front 2024-04-13T03:53:06.831392+0000 (oldest deadline 2024-04-13T03:53:32.131543+0000) 2024-04-13T03:53:33.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:32 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:32.778+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:33.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:32 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:32.778+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:33.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:32 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:32.778+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:33.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:32 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:32.778+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:33.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:32 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:32.778+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:33.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:32 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:32.778+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:33.326 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:32.941+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:33.326 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:32.941+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:33.326 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:32.941+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:33.327 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:32.941+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:33.327 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:32 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:32.941+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:33.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:32.923+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:33.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:32.923+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:33.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:32.923+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:33.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:32.923+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:33.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:32.923+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:33.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:32 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:32.923+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:33.610 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:33 smithi188 podman[76600]: 2024-04-13 03:53:33.326848119 +0000 UTC m=+5.515237558 container died 64d3d1ae48ead118df11ba6049a5436671c845ebc820bf7dcb1263298aabab40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-13T03:53:33.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:33 smithi188 podman[76600]: 2024-04-13 03:53:33.702475173 +0000 UTC m=+5.890864596 container cleanup 64d3d1ae48ead118df11ba6049a5436671c845ebc820bf7dcb1263298aabab40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-13T03:53:33.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:33 smithi188 bash[76600]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8 2024-04-13T03:53:34.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:33 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:33.795+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:34.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:33 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:33.795+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:34.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:33 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:33.795+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:34.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:33 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:33.795+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:34.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:33 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:33.795+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:34.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:33 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:33.795+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:34.312 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:33 smithi188 podman[76617]: 2024-04-13 03:53:33.97902507 +0000 UTC m=+0.646811735 container remove 64d3d1ae48ead118df11ba6049a5436671c845ebc820bf7dcb1263298aabab40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-13T03:53:34.313 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:33 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:33.924+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:34.313 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:33 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:33.924+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:34.313 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:33 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:33.924+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:34.313 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:33 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:33.924+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:34.313 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:33 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:33.924+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:34.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:33.945+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:34.356 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:33.945+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:34.356 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:33.945+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:34.356 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:33.945+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:34.356 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:33.945+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:34.357 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:33 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:33.945+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:34.607 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:34 smithi188 podman[76707]: 2024-04-13 03:53:34.216274598 +0000 UTC m=+0.017914363 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:53:34.607 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:34 smithi188 podman[76707]: 2024-04-13 03:53:34.329712304 +0000 UTC m=+0.131352064 container create 4cc6651629905f4af975bab641a2c4ad0156383cb18ae9c86b414e16c64ed991 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-13T03:53:34.872 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:34 smithi188 podman[76707]: 2024-04-13 03:53:34.607587304 +0000 UTC m=+0.409227061 container init 4cc6651629905f4af975bab641a2c4ad0156383cb18ae9c86b414e16c64ed991 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-13T03:53:34.873 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:34 smithi188 podman[76707]: 2024-04-13 03:53:34.610730048 +0000 UTC m=+0.412369805 container start 4cc6651629905f4af975bab641a2c4ad0156383cb18ae9c86b414e16c64ed991 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:53:34.873 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:34 smithi188 podman[76707]: 2024-04-13 03:53:34.699191278 +0000 UTC m=+0.500831044 container attach 4cc6651629905f4af975bab641a2c4ad0156383cb18ae9c86b414e16c64ed991 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:53:35.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:34 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:34.814+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:35.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:34 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:34.814+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:35.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:34 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:34.814+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:35.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:34 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:34.814+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:35.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:34 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:34.814+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:35.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:34 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:34.814+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:35.168 INFO:journalctl@ceph.osd.8.smithi188.stdout:Apr 13 03:53:34 smithi188 podman[76707]: 2024-04-13 03:53:34.899493655 +0000 UTC m=+0.701133421 container died 4cc6651629905f4af975bab641a2c4ad0156383cb18ae9c86b414e16c64ed991 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-13T03:53:35.168 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:34 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:34.952+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:35.168 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:34 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:34.952+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:35.169 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:34 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:34.952+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:35.169 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:34 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:34.952+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:35.169 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:34 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:34.952+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:35.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:34 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:34.983+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:35.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:34 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:34.983+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:35.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:34 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:34.983+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:35.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:34 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:34.983+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:35.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:34 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:34.983+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:35.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:34 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:34.983+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:35.450 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.8.service' 2024-04-13T03:53:35.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:53:35.531 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-13T03:53:35.531 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-13T03:53:35.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.9 2024-04-13T03:53:36.012 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:35 smithi016 systemd[1]: Stopping Ceph osd.9 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:53:36.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:35 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:35.775+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:36.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:35 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:35.775+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:36.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:35 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:35.775+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:36.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:35 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:35.775+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:36.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:35 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:35.775+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:36.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:35 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:35.775+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:36.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.010+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:36.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.010+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:36.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.010+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:36.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.010+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:36.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.010+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:36.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.010+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:36.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.193+0000 7ff2eb600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:53:36.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.194+0000 7ff2eb600700 -1 osd.9 121 *** Got signal Terminated *** 2024-04-13T03:53:36.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.194+0000 7ff2eb600700 -1 osd.9 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:53:36.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:35 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:35.978+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:36.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:35 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:35.978+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:36.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:35 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:35.978+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:36.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:35 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:35.978+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:36.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:35 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:35.978+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:37.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:36 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:36.820+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:37.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:36 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:36.820+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:37.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:36 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:36.820+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:37.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:36 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:36.820+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:37.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:36 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:36.820+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:37.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:36 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:36.820+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:37.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.975+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:37.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.975+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:37.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.975+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:37.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.975+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:37.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.975+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:37.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:36 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:36.975+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:37.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:37.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:37.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:37.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:37.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:37.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:37.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:37.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:37.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:37.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:38.320 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:37.856+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:38.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:37.856+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:38.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:37.856+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:38.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:37.856+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:38.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:37.856+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:38.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:37.856+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:38.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:37 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:37.856+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:38.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:38.024+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:38.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:38.024+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:38.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:38.024+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:38.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:38.024+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:38.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:38.024+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:38.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:38 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:38.024+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:38.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:38 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:38.058+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:38.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:38 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:38.058+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:38.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:38 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:38.058+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:38.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:38 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:38.058+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:38.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:38 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:38.058+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:39.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:38 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:38.812+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:39.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:38 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:38.812+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:39.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:38 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:38.812+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:39.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:38 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:38.812+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:39.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:38 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:38.812+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:39.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:38 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:38.812+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:39.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:38 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:38.812+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:39.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:39.039+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:39.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:39.039+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:39.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:39.039+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:39.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:39.039+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:39.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:39.039+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:39.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:39 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:39.039+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:39.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:39.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:39.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:39.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:39.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.024+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:40.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:39.779+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:40.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:39.779+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:40.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:39.779+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:40.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:39.779+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:40.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:39.779+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:40.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:39.779+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:40.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:39 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:39.779+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:40.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:40 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:40.013+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:40.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:40 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:40.013+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:40.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:40 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:40.013+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:40.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:40 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:40.013+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:40.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:40 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:40.013+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:40.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:40 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:40.013+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:40.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.990+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:40.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.990+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:40.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.990+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:40.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.990+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:40.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:39 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:39.990+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:41.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:40.734+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:41.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:40.734+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:41.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:40.734+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:41.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:40.734+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:41.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:40.734+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:41.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:40.734+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:41.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:40 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:40.734+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:41.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:41.052+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.280129+0000 front 2024-04-13T03:52:23.280197+0000 (oldest deadline 2024-04-13T03:52:45.580400+0000) 2024-04-13T03:53:41.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:41.052+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:31.385940+0000 front 2024-04-13T03:52:31.386044+0000 (oldest deadline 2024-04-13T03:52:56.686179+0000) 2024-04-13T03:53:41.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:41.052+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:36.687615+0000 front 2024-04-13T03:52:36.687688+0000 (oldest deadline 2024-04-13T03:53:02.587875+0000) 2024-04-13T03:53:41.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:41.052+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.090880+0000 front 2024-04-13T03:52:46.091061+0000 (oldest deadline 2024-04-13T03:53:11.991104+0000) 2024-04-13T03:53:41.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:41.052+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:57.195591+0000 front 2024-04-13T03:52:57.195519+0000 (oldest deadline 2024-04-13T03:53:23.095784+0000) 2024-04-13T03:53:41.355 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:41 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9[64026]: 2024-04-13T03:53:41.052+0000 7ff2e1e00700 -1 osd.9 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:03.598728+0000 front 2024-04-13T03:53:03.598661+0000 (oldest deadline 2024-04-13T03:53:29.498924+0000) 2024-04-13T03:53:41.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:41 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:41.027+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:41.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:41 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:41.027+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:41.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:41 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:41.027+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:41.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:41 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:41.027+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:41.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:41 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:41.027+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:41.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:41 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:41.027+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:42.064 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:41 smithi016 podman[92177]: 2024-04-13 03:53:41.804504288 +0000 UTC m=+5.801756206 container died 572a1f28c1645c94bb107b664c1afb4a6c43eeae8c57829c4f5956f0cd53fa45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD) 2024-04-13T03:53:42.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:41 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:41.708+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:42.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:41 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:41.708+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:42.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:41 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:41.708+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:42.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:41 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:41.708+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:42.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:41 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:41.708+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:42.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:41 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:41.708+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:42.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:41 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:41.708+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:42.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:42 smithi016 podman[92177]: 2024-04-13 03:53:42.195704683 +0000 UTC m=+6.192956594 container cleanup 572a1f28c1645c94bb107b664c1afb4a6c43eeae8c57829c4f5956f0cd53fa45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-13T03:53:42.354 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:42 smithi016 bash[92177]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9 2024-04-13T03:53:42.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:42 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:42.068+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:42.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:42 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:42.068+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:42.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:42 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:42.068+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:42.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:42 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:42.068+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:42.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:42 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:42.068+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:42.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:42 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:42.068+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:42.853 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:42 smithi016 podman[92195]: 2024-04-13 03:53:42.374048616 +0000 UTC m=+0.561093711 container remove 572a1f28c1645c94bb107b664c1afb4a6c43eeae8c57829c4f5956f0cd53fa45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-04-13T03:53:43.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:42.750+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:43.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:42.750+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:43.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:42.750+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:43.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:42.750+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:43.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:42.750+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:43.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:42.750+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:43.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:42 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:42.750+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:43.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:43 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:43.077+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:43.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:43 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:43.077+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:43.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:43 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:43.077+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:43.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:43 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:43.077+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:43.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:43 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:43.077+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:43.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:43 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:43.077+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:43.458 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:43 smithi016 podman[92279]: 2024-04-13 03:53:43.101033213 +0000 UTC m=+0.037917537 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:53:43.458 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:43 smithi016 podman[92279]: 2024-04-13 03:53:43.289649773 +0000 UTC m=+0.226534122 container create 4cf390e6ad71dbecba1e0fe33cfc3fa54cd42180c28812b999ba36f833c9a648 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-13T03:53:44.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:43 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:43.742+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:44.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:43 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:43.742+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:44.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:43 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:43.742+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:44.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:43 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:43.742+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:44.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:43 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:43.742+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:44.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:43 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:43.742+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:44.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:43 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:43.743+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:44.099 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:43 smithi016 podman[92279]: 2024-04-13 03:53:43.618163229 +0000 UTC m=+0.555047585 container init 4cf390e6ad71dbecba1e0fe33cfc3fa54cd42180c28812b999ba36f833c9a648 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:53:44.100 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:43 smithi016 podman[92279]: 2024-04-13 03:53:43.629817601 +0000 UTC m=+0.566701917 container start 4cf390e6ad71dbecba1e0fe33cfc3fa54cd42180c28812b999ba36f833c9a648 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS) 2024-04-13T03:53:44.100 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:43 smithi016 podman[92279]: 2024-04-13 03:53:43.700200938 +0000 UTC m=+0.637085317 container attach 4cf390e6ad71dbecba1e0fe33cfc3fa54cd42180c28812b999ba36f833c9a648 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:53:44.100 INFO:journalctl@ceph.osd.9.smithi016.stdout:Apr 13 03:53:44 smithi016 podman[92279]: 2024-04-13 03:53:44.097627058 +0000 UTC m=+1.034511375 container died 4cf390e6ad71dbecba1e0fe33cfc3fa54cd42180c28812b999ba36f833c9a648 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:53:44.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:44 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:44.113+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:44.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:44 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:44.113+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:44.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:44 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:44.113+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:44.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:44 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:44.113+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:44.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:44 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:44.113+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:44.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:44 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:44.113+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:44.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.9.service' 2024-04-13T03:53:44.971 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:53:44.972 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-13T03:53:44.972 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-13T03:53:44.972 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.10 2024-04-13T03:53:45.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:44 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:44.773+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:45.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:44 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:44.773+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:45.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:44 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:44.773+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:45.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:44 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:44.773+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:45.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:44 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:44.773+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:45.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:44 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:44.773+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:45.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:44 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:44.773+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:45.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 systemd[1]: Stopping Ceph osd.10 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:53:45.570 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.251+0000 7f477b291700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:53:45.571 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.251+0000 7f477b291700 -1 osd.10 121 *** Got signal Terminated *** 2024-04-13T03:53:45.571 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.251+0000 7f477b291700 -1 osd.10 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:53:45.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:45 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:45.132+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:45.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:45 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:45.132+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:45.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:45 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:45.132+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:45.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:45 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:45.132+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:45.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:45 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:45.132+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:45.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:45 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:45.132+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:46.019 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.762+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:46.019 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.762+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:46.020 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.762+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:46.020 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.762+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:46.020 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.762+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:46.020 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.762+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:46.020 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:45 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:45.762+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:46.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:46 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:46.182+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:46.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:46 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:46.182+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:46.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:46 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:46.182+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:46.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:46 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:46.182+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:46.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:46 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:46.182+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:46.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:46 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:46.182+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:47.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:46 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:46.803+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:47.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:46 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:46.803+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:47.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:46 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:46.803+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:47.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:46 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:46.803+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:47.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:46 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:46.803+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:47.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:46 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:46.803+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:47.072 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:46 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:46.804+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:47.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:47 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:47.222+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:47.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:47 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:47.222+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:47.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:47 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:47.222+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:47.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:47 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:47.222+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:47.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:47 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:47.222+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:47.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:47 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:47.222+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:48.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:47 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:47.822+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:48.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:47 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:47.822+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:48.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:47 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:47.822+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:48.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:47 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:47.822+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:48.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:47 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:47.822+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:48.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:47 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:47.822+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:48.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:47 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:47.823+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:48.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:48 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:48.225+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:48.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:48 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:48.225+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:48.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:48 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:48.225+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:48.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:48 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:48.225+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:48.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:48 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:48.225+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:48.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:48 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:48.225+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:48.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:48 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:48.225+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:49.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:48 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:48.781+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:49.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:48 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:48.781+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:49.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:48 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:48.781+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:49.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:48 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:48.781+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:49.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:48 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:48.781+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:49.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:48 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:48.781+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:49.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:48 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:48.781+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:49.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:49 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:49.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:49.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:49 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:49.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:49.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:49 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:49.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:49.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:49 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:49.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:49.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:49 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:49.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:49.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:49 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:49.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:49.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:49 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:49.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:50.070 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:49 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:49.810+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:23.018926+0000 front 2024-04-13T03:52:23.018754+0000 (oldest deadline 2024-04-13T03:52:47.118719+0000) 2024-04-13T03:53:50.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:49 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:49.810+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:28.819707+0000 front 2024-04-13T03:52:28.819685+0000 (oldest deadline 2024-04-13T03:52:54.719887+0000) 2024-04-13T03:53:50.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:49 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:49.810+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.721241+0000 front 2024-04-13T03:52:41.721355+0000 (oldest deadline 2024-04-13T03:53:06.322073+0000) 2024-04-13T03:53:50.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:49 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:49.810+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6814 osd.3 since back 2024-04-13T03:52:46.322805+0000 front 2024-04-13T03:52:46.322912+0000 (oldest deadline 2024-04-13T03:53:11.022645+0000) 2024-04-13T03:53:50.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:49 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:49.810+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.323864+0000 front 2024-04-13T03:52:56.323902+0000 (oldest deadline 2024-04-13T03:53:22.224132+0000) 2024-04-13T03:53:50.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:49 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:49.810+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:02.224670+0000 front 2024-04-13T03:53:02.224744+0000 (oldest deadline 2024-04-13T03:53:31.625334+0000) 2024-04-13T03:53:50.071 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:49 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10[60568]: 2024-04-13T03:53:49.810+0000 7f477387d700 -1 osd.10 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:11.626007+0000 front 2024-04-13T03:53:11.626143+0000 (oldest deadline 2024-04-13T03:53:37.525729+0000) 2024-04-13T03:53:50.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:50 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:50.178+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:50.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:50 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:50.178+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:50.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:50 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:50.178+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:50.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:50 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:50.178+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:50.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:50 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:50.178+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:50.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:50 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:50.178+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:50.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:50 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:50.178+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:50.998 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:50 smithi119 podman[74508]: 2024-04-13 03:53:50.707013769 +0000 UTC m=+5.542656072 container died 664bcdb4be1a8c88e0c46ea39d6a132c55c350a52c6b69da2e651d49f02b6c87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:53:51.320 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:51 smithi119 podman[74508]: 2024-04-13 03:53:51.085503696 +0000 UTC m=+5.921145999 container cleanup 664bcdb4be1a8c88e0c46ea39d6a132c55c350a52c6b69da2e651d49f02b6c87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-13T03:53:51.321 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:51 smithi119 bash[74508]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10 2024-04-13T03:53:51.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:51 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:51.204+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:51.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:51 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:51.204+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:51.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:51 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:51.204+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:51.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:51 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:51.204+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:51.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:51 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:51.204+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:51.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:51 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:51.204+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:51.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:51 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:51.204+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:51.645 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:51 smithi119 podman[74525]: 2024-04-13 03:53:51.328193505 +0000 UTC m=+0.616491460 container remove 664bcdb4be1a8c88e0c46ea39d6a132c55c350a52c6b69da2e651d49f02b6c87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:53:51.897 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:51 smithi119 podman[74620]: 2024-04-13 03:53:51.549133015 +0000 UTC m=+0.017682797 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:53:51.898 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:51 smithi119 podman[74620]: 2024-04-13 03:53:51.653909616 +0000 UTC m=+0.122459392 container create a9b7df3cdc2def5d9b72e11bd6891de99fac394974e6be4678afb3d605996f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:53:51.898 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:51 smithi119 podman[74620]: 2024-04-13 03:53:51.898217191 +0000 UTC m=+0.366766967 container init a9b7df3cdc2def5d9b72e11bd6891de99fac394974e6be4678afb3d605996f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-13T03:53:52.184 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:51 smithi119 podman[74620]: 2024-04-13 03:53:51.901360407 +0000 UTC m=+0.369910182 container start a9b7df3cdc2def5d9b72e11bd6891de99fac394974e6be4678afb3d605996f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:53:52.347 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:51 smithi119 podman[74620]: 2024-04-13 03:53:51.989817256 +0000 UTC m=+0.458367058 container attach a9b7df3cdc2def5d9b72e11bd6891de99fac394974e6be4678afb3d605996f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:53:52.476 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:52 smithi119 podman[74620]: 2024-04-13 03:53:52.18528616 +0000 UTC m=+0.653835950 container died a9b7df3cdc2def5d9b72e11bd6891de99fac394974e6be4678afb3d605996f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:53:52.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:52 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:52.187+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:52.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:52 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:52.187+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:52.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:52 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:52.187+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:52.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:52 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:52.187+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:52.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:52 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:52.187+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:52.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:52 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:52.187+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:52.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:52 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:52.187+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:52.741 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 13 03:53:52 smithi119 podman[74646]: 2024-04-13 03:53:52.720368047 +0000 UTC m=+0.530245935 container remove a9b7df3cdc2def5d9b72e11bd6891de99fac394974e6be4678afb3d605996f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-13T03:53:52.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.10.service' 2024-04-13T03:53:52.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:53:52.793 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-13T03:53:52.793 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-13T03:53:52.793 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.11 2024-04-13T03:53:53.087 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:52 smithi188 systemd[1]: Stopping Ceph osd.11 for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:53:53.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.087+0000 7f5892c5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:53:53.372 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.087+0000 7f5892c5e700 -1 osd.11 121 *** Got signal Terminated *** 2024-04-13T03:53:53.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.087+0000 7f5892c5e700 -1 osd.11 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-13T03:53:53.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:53.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:53.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:53.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:53.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:53.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:53.373 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:53 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:53.214+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:54.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:54.254+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:54.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:54.254+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:54.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:54.254+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:54.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:54.254+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:54.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:54.254+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:54.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:54.254+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:54.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:54 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:54.254+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:55.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:55.267+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:55.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:55.267+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:55.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:55.267+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:55.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:55.267+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:55.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:55.267+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:55.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:55.267+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:55.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:55.267+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:55.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:55 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:55.267+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2024-04-13T03:53:30.801749+0000 front 2024-04-13T03:53:30.801713+0000 (oldest deadline 2024-04-13T03:53:54.901754+0000) 2024-04-13T03:53:56.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:56.303+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:56.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:56.303+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:56.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:56.303+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:56.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:56.303+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:56.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:56.303+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:56.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:56.303+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:56.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:56.303+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:56.624 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:56 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:56.303+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2024-04-13T03:53:30.801749+0000 front 2024-04-13T03:53:30.801713+0000 (oldest deadline 2024-04-13T03:53:54.901754+0000) 2024-04-13T03:53:57.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:57.339+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-13T03:52:20.691704+0000 front 2024-04-13T03:52:20.691546+0000 (oldest deadline 2024-04-13T03:52:50.692258+0000) 2024-04-13T03:53:57.622 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:57.339+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-13T03:52:30.692919+0000 front 2024-04-13T03:52:30.692763+0000 (oldest deadline 2024-04-13T03:52:54.792934+0000) 2024-04-13T03:53:57.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:57.339+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6804 osd.2 since back 2024-04-13T03:52:41.095524+0000 front 2024-04-13T03:52:41.095835+0000 (oldest deadline 2024-04-13T03:53:06.395711+0000) 2024-04-13T03:53:57.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:57.339+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-13T03:52:56.297529+0000 front 2024-04-13T03:52:56.297598+0000 (oldest deadline 2024-04-13T03:53:25.697916+0000) 2024-04-13T03:53:57.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:57.339+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6812 osd.5 since back 2024-04-13T03:53:05.698397+0000 front 2024-04-13T03:53:05.698448+0000 (oldest deadline 2024-04-13T03:53:28.598454+0000) 2024-04-13T03:53:57.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:57.339+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.16:6822 osd.6 since back 2024-04-13T03:53:14.499689+0000 front 2024-04-13T03:53:14.499624+0000 (oldest deadline 2024-04-13T03:53:40.299837+0000) 2024-04-13T03:53:57.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:57.339+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-13T03:53:20.300736+0000 front 2024-04-13T03:53:20.300683+0000 (oldest deadline 2024-04-13T03:53:47.901009+0000) 2024-04-13T03:53:57.623 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:57 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11[61962]: 2024-04-13T03:53:57.339+0000 7f588f252700 -1 osd.11 121 heartbeat_check: no reply from 172.21.15.188:6820 osd.8 since back 2024-04-13T03:53:30.801749+0000 front 2024-04-13T03:53:30.801713+0000 (oldest deadline 2024-04-13T03:53:54.901754+0000) 2024-04-13T03:53:58.912 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:58 smithi188 podman[76827]: 2024-04-13 03:53:58.640950547 +0000 UTC m=+5.655812377 container died 93f9e9735781f9976d12187607e288aecb3f67a92a3c258a116fc9364939fb83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-13T03:53:59.262 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:58 smithi188 podman[76827]: 2024-04-13 03:53:58.995295707 +0000 UTC m=+6.010157493 container cleanup 93f9e9735781f9976d12187607e288aecb3f67a92a3c258a116fc9364939fb83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:53:59.262 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:58 smithi188 bash[76827]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11 2024-04-13T03:53:59.596 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:59 smithi188 podman[76844]: 2024-04-13 03:53:59.263039063 +0000 UTC m=+0.612971182 container remove 93f9e9735781f9976d12187607e288aecb3f67a92a3c258a116fc9364939fb83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-13T03:53:59.857 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:59 smithi188 podman[76942]: 2024-04-13 03:53:59.597065081 +0000 UTC m=+0.113400245 container create ca92cc64932e05c19bb63c95f6c17433102cbfa70f85476f191ae3a400523f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-13T03:53:59.857 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:59 smithi188 podman[76942]: 2024-04-13 03:53:59.501502891 +0000 UTC m=+0.017838071 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-13T03:53:59.857 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:59 smithi188 podman[76942]: 2024-04-13 03:53:59.783560273 +0000 UTC m=+0.299895453 container init ca92cc64932e05c19bb63c95f6c17433102cbfa70f85476f191ae3a400523f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS) 2024-04-13T03:53:59.857 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:59 smithi188 podman[76942]: 2024-04-13 03:53:59.786877532 +0000 UTC m=+0.303212702 container start ca92cc64932e05c19bb63c95f6c17433102cbfa70f85476f191ae3a400523f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-13T03:53:59.857 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:53:59 smithi188 podman[76942]: 2024-04-13 03:53:59.857555539 +0000 UTC m=+0.373890722 container attach ca92cc64932e05c19bb63c95f6c17433102cbfa70f85476f191ae3a400523f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-13T03:54:00.122 INFO:journalctl@ceph.osd.11.smithi188.stdout:Apr 13 03:54:00 smithi188 podman[76942]: 2024-04-13 03:54:00.076657638 +0000 UTC m=+0.592992806 container died ca92cc64932e05c19bb63c95f6c17433102cbfa70f85476f191ae3a400523f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-13T03:54:00.656 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@osd.11.service' 2024-04-13T03:54:00.696 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:00.696 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-13T03:54:00.697 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-13T03:54:00.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.a 2024-04-13T03:54:01.103 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 13 03:54:00 smithi016 systemd[1]: Stopping Ceph mds.a for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:01.606 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 13 03:54:01 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-a[70563]: 2024-04-13T03:54:01.303+0000 7feefa200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:01.606 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 13 03:54:01 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-a[70563]: 2024-04-13T03:54:01.303+0000 7feefa200700 -1 mds.a *** got signal Terminated *** 2024-04-13T03:54:04.724 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 13 03:54:04 smithi016 podman[92384]: 2024-04-13 03:54:04.361837919 +0000 UTC m=+3.191646443 container stop bb75f63a4c632afa9d2fcbd60701191987f7ea2cf7350155c0b2685e9c41c02c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:54:04.724 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 13 03:54:04 smithi016 podman[92384]: 2024-04-13 03:54:04.388331708 +0000 UTC m=+3.218140187 container died bb75f63a4c632afa9d2fcbd60701191987f7ea2cf7350155c0b2685e9c41c02c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-13T03:54:05.028 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 13 03:54:04 smithi016 podman[92384]: 2024-04-13 03:54:04.885863808 +0000 UTC m=+3.715672325 container cleanup bb75f63a4c632afa9d2fcbd60701191987f7ea2cf7350155c0b2685e9c41c02c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:54:05.029 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 13 03:54:04 smithi016 bash[92384]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-a 2024-04-13T03:54:05.353 INFO:journalctl@ceph.mds.a.smithi016.stdout:Apr 13 03:54:05 smithi016 podman[92410]: 2024-04-13 03:54:05.028724274 +0000 UTC m=+0.645539364 container remove bb75f63a4c632afa9d2fcbd60701191987f7ea2cf7350155c0b2685e9c41c02c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-13T03:54:05.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.a.service' 2024-04-13T03:54:05.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:05.636 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-13T03:54:05.636 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-13T03:54:05.637 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.d 2024-04-13T03:54:06.103 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:54:05 smithi016 systemd[1]: Stopping Ceph mds.d for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:06.854 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:54:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d[69557]: 2024-04-13T03:54:06.490+0000 7ff908600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:06.854 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:54:06 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d[69557]: 2024-04-13T03:54:06.490+0000 7ff908600700 -1 mds.d *** got signal Terminated *** 2024-04-13T03:54:08.770 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:54:08 smithi016 podman[92516]: 2024-04-13 03:54:08.411508096 +0000 UTC m=+2.281830180 container died 050a364f12310c57d2dc308c1293a3eecbc1476f4493c933f8ec04a10c655c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-13T03:54:09.092 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:54:08 smithi016 podman[92516]: 2024-04-13 03:54:08.881841641 +0000 UTC m=+2.752163729 container cleanup 050a364f12310c57d2dc308c1293a3eecbc1476f4493c933f8ec04a10c655c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409) 2024-04-13T03:54:09.093 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:54:08 smithi016 bash[92516]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d 2024-04-13T03:54:09.353 INFO:journalctl@ceph.mds.d.smithi016.stdout:Apr 13 03:54:09 smithi016 podman[92530]: 2024-04-13 03:54:09.091077496 +0000 UTC m=+0.664667709 container remove 050a364f12310c57d2dc308c1293a3eecbc1476f4493c933f8ec04a10c655c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-13T03:54:09.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.d.service' 2024-04-13T03:54:09.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:09.784 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-13T03:54:09.784 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-13T03:54:09.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.g 2024-04-13T03:54:10.354 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:54:09 smithi016 systemd[1]: Stopping Ceph mds.g for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:10.853 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:54:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g[70182]: 2024-04-13T03:54:10.367+0000 7f7ec8a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:10.854 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:54:10 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g[70182]: 2024-04-13T03:54:10.367+0000 7f7ec8a00700 -1 mds.g *** got signal Terminated *** 2024-04-13T03:54:12.354 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:54:12 smithi016 podman[92636]: 2024-04-13 03:54:12.089664497 +0000 UTC m=+1.858656051 container stop 8fc36c5f69e5073d0fa3e8b770c2a6d97b37655c3f9f0cdd71d66a7a44814fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.build-date=20240409) 2024-04-13T03:54:12.354 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:54:12 smithi016 podman[92636]: 2024-04-13 03:54:12.11608157 +0000 UTC m=+1.885073087 container died 8fc36c5f69e5073d0fa3e8b770c2a6d97b37655c3f9f0cdd71d66a7a44814fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:54:12.805 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:54:12 smithi016 podman[92636]: 2024-04-13 03:54:12.543643966 +0000 UTC m=+2.312635513 container cleanup 8fc36c5f69e5073d0fa3e8b770c2a6d97b37655c3f9f0cdd71d66a7a44814fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-13T03:54:12.806 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:54:12 smithi016 bash[92636]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g 2024-04-13T03:54:13.103 INFO:journalctl@ceph.mds.g.smithi016.stdout:Apr 13 03:54:12 smithi016 podman[92649]: 2024-04-13 03:54:12.803019843 +0000 UTC m=+0.694532383 container remove 8fc36c5f69e5073d0fa3e8b770c2a6d97b37655c3f9f0cdd71d66a7a44814fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-13T03:54:13.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.g.service' 2024-04-13T03:54:13.434 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:13.434 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-13T03:54:13.435 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-13T03:54:13.435 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.j 2024-04-13T03:54:13.853 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:54:13 smithi016 systemd[1]: Stopping Ceph mds.j for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:14.354 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:54:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j[69190]: 2024-04-13T03:54:14.078+0000 7ff14c400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:14.354 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:54:14 smithi016 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j[69190]: 2024-04-13T03:54:14.078+0000 7ff14c400700 -1 mds.j *** got signal Terminated *** 2024-04-13T03:54:16.778 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:54:16 smithi016 podman[92755]: 2024-04-13 03:54:16.424747261 +0000 UTC m=+2.483052165 container died 1495192ece6d0a588285f70dd8680aec62b9c0eb4464e2a85f533169a9e7d488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-13T03:54:17.103 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:54:16 smithi016 podman[92755]: 2024-04-13 03:54:16.881274449 +0000 UTC m=+2.939579384 container cleanup 1495192ece6d0a588285f70dd8680aec62b9c0eb4464e2a85f533169a9e7d488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:54:17.104 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:54:16 smithi016 bash[92755]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j 2024-04-13T03:54:17.603 INFO:journalctl@ceph.mds.j.smithi016.stdout:Apr 13 03:54:17 smithi016 podman[92780]: 2024-04-13 03:54:17.115347906 +0000 UTC m=+0.671627848 container remove 1495192ece6d0a588285f70dd8680aec62b9c0eb4464e2a85f533169a9e7d488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-j, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409) 2024-04-13T03:54:17.660 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.j.service' 2024-04-13T03:54:17.782 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:17.783 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-13T03:54:17.783 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-13T03:54:17.783 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.b 2024-04-13T03:54:18.071 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:54:17 smithi119 systemd[1]: Stopping Ceph mds.b for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:18.340 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:54:18 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: 2024-04-13T03:54:18.070+0000 7f8faacbf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:18.340 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:54:18 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b[66222]: 2024-04-13T03:54:18.070+0000 7f8faacbf700 -1 mds.b *** got signal Terminated *** 2024-04-13T03:54:20.662 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:54:20 smithi119 podman[74725]: 2024-04-13 03:54:20.364203711 +0000 UTC m=+2.388977898 container died d80ab5e311a12f00a3b9576118b3ebc911cb90b5bbc8870aa02370f450d27f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-13T03:54:20.937 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:54:20 smithi119 podman[74725]: 2024-04-13 03:54:20.737074657 +0000 UTC m=+2.761848830 container cleanup d80ab5e311a12f00a3b9576118b3ebc911cb90b5bbc8870aa02370f450d27f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_BRANCH=HEAD) 2024-04-13T03:54:20.937 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 13 03:54:20 smithi119 bash[74725]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-b 2024-04-13T03:54:21.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.b.service' 2024-04-13T03:54:21.186 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:21.186 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-13T03:54:21.186 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-13T03:54:21.186 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.e 2024-04-13T03:54:21.507 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:54:21 smithi119 systemd[1]: Stopping Ceph mds.e for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:21.820 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:54:21 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-e[65609]: 2024-04-13T03:54:21.507+0000 7f9dc78e2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:21.821 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:54:21 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-e[65609]: 2024-04-13T03:54:21.507+0000 7f9dc78e2700 -1 mds.e *** got signal Terminated *** 2024-04-13T03:54:24.681 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:54:24 smithi119 podman[74858]: 2024-04-13 03:54:24.367264242 +0000 UTC m=+2.958524109 container died ae69677435c8d86082e28127571769c4db125c97b2b2caf2f8bc5c53dfb3e773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-e, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-13T03:54:24.681 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:54:24 smithi119 podman[74858]: 2024-04-13 03:54:24.682146587 +0000 UTC m=+3.273406414 container cleanup ae69677435c8d86082e28127571769c4db125c97b2b2caf2f8bc5c53dfb3e773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-e, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409) 2024-04-13T03:54:25.070 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:54:24 smithi119 bash[74858]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-e 2024-04-13T03:54:25.071 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 13 03:54:24 smithi119 podman[74872]: 2024-04-13 03:54:24.899858836 +0000 UTC m=+0.533011039 container remove ae69677435c8d86082e28127571769c4db125c97b2b2caf2f8bc5c53dfb3e773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-e, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409) 2024-04-13T03:54:25.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.e.service' 2024-04-13T03:54:25.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:25.153 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-13T03:54:25.154 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-13T03:54:25.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.h 2024-04-13T03:54:25.468 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:54:25 smithi119 systemd[1]: Stopping Ceph mds.h for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:25.820 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:54:25 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h[65914]: 2024-04-13T03:54:25.468+0000 7f3ba23df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:25.821 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:54:25 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h[65914]: 2024-04-13T03:54:25.468+0000 7f3ba23df700 -1 mds.h *** got signal Terminated *** 2024-04-13T03:54:28.646 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:54:28 smithi119 podman[74978]: 2024-04-13 03:54:28.348511902 +0000 UTC m=+2.982871805 container died cb36ac31d9928920e7ef19c9b148f8be43256f352f6b260e6ff678c42888674b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.schema-version=1.0) 2024-04-13T03:54:28.993 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:54:28 smithi119 podman[74978]: 2024-04-13 03:54:28.710064323 +0000 UTC m=+3.344424226 container cleanup cb36ac31d9928920e7ef19c9b148f8be43256f352f6b260e6ff678c42888674b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-13T03:54:28.994 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 13 03:54:28 smithi119 bash[74978]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-h 2024-04-13T03:54:29.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.h.service' 2024-04-13T03:54:29.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:29.176 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-13T03:54:29.176 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-13T03:54:29.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.k 2024-04-13T03:54:29.504 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:54:29 smithi119 systemd[1]: Stopping Ceph mds.k for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:29.820 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:54:29 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k[65292]: 2024-04-13T03:54:29.504+0000 7f641f6cf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:29.820 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:54:29 smithi119 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k[65292]: 2024-04-13T03:54:29.504+0000 7f641f6cf700 -1 mds.k *** got signal Terminated *** 2024-04-13T03:54:32.713 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:54:32 smithi119 podman[75099]: 2024-04-13 03:54:32.354178043 +0000 UTC m=+2.955614079 container died 4e18168fa20f0af1c16cfe2f81fd15cfdfd4a9beaa70c4e013a83e9d84fd331f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-13T03:54:33.055 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:54:32 smithi119 podman[75099]: 2024-04-13 03:54:32.80525288 +0000 UTC m=+3.406688919 container cleanup 4e18168fa20f0af1c16cfe2f81fd15cfdfd4a9beaa70c4e013a83e9d84fd331f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:54:33.055 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 13 03:54:32 smithi119 bash[75099]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-k 2024-04-13T03:54:33.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.k.service' 2024-04-13T03:54:33.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:33.195 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-13T03:54:33.195 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-13T03:54:33.195 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.c 2024-04-13T03:54:33.549 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:54:33 smithi188 systemd[1]: Stopping Ceph mds.c for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:33.872 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:54:33 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c[67425]: 2024-04-13T03:54:33.548+0000 7f3d9a3cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:33.872 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:54:33 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c[67425]: 2024-04-13T03:54:33.548+0000 7f3d9a3cc700 -1 mds.c *** got signal Terminated *** 2024-04-13T03:54:36.622 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:54:36 smithi188 podman[77050]: 2024-04-13 03:54:36.355455134 +0000 UTC m=+2.968232282 container died ea3859d0b276d4d7e081600838cee7691d27e1397373de852d717f79d0b8d7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-13T03:54:37.066 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:54:36 smithi188 podman[77050]: 2024-04-13 03:54:36.832301209 +0000 UTC m=+3.445078361 container cleanup ea3859d0b276d4d7e081600838cee7691d27e1397373de852d717f79d0b8d7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409) 2024-04-13T03:54:37.066 INFO:journalctl@ceph.mds.c.smithi188.stdout:Apr 13 03:54:36 smithi188 bash[77050]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-c 2024-04-13T03:54:37.201 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.c.service' 2024-04-13T03:54:37.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:37.239 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-13T03:54:37.239 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-13T03:54:37.239 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.f 2024-04-13T03:54:37.622 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:54:37 smithi188 systemd[1]: Stopping Ceph mds.f for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:37.622 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:54:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f[67114]: 2024-04-13T03:54:37.561+0000 7ff7c9bb4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:37.622 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:54:37 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f[67114]: 2024-04-13T03:54:37.561+0000 7ff7c9bb4700 -1 mds.f *** got signal Terminated *** 2024-04-13T03:54:40.622 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:54:40 smithi188 podman[77182]: 2024-04-13 03:54:40.361222753 +0000 UTC m=+2.903021449 container died 2e1bc20a2d707fe275f8c8ced94e86783c56f89d9ebfecf4a0f73954a918b2c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-13T03:54:40.944 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:54:40 smithi188 podman[77182]: 2024-04-13 03:54:40.752447762 +0000 UTC m=+3.294246431 container cleanup 2e1bc20a2d707fe275f8c8ced94e86783c56f89d9ebfecf4a0f73954a918b2c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-13T03:54:40.944 INFO:journalctl@ceph.mds.f.smithi188.stdout:Apr 13 03:54:40 smithi188 bash[77182]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-f 2024-04-13T03:54:41.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.f.service' 2024-04-13T03:54:41.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:41.110 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-13T03:54:41.110 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-13T03:54:41.110 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.i 2024-04-13T03:54:41.372 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:54:41 smithi188 systemd[1]: Stopping Ceph mds.i for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:41.872 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:54:41 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i[67734]: 2024-04-13T03:54:41.422+0000 7f71556bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:41.872 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:54:41 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i[67734]: 2024-04-13T03:54:41.422+0000 7f71556bd700 -1 mds.i *** got signal Terminated *** 2024-04-13T03:54:44.622 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:54:44 smithi188 podman[77302]: 2024-04-13 03:54:44.363010337 +0000 UTC m=+3.038239860 container died cc5aff15da9c5d83b4a74f501f89ebee995a1c616622e77105e0b641e476da30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-13T03:54:45.156 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:54:44 smithi188 podman[77302]: 2024-04-13 03:54:44.97205178 +0000 UTC m=+3.647281281 container cleanup cc5aff15da9c5d83b4a74f501f89ebee995a1c616622e77105e0b641e476da30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-13T03:54:45.156 INFO:journalctl@ceph.mds.i.smithi188.stdout:Apr 13 03:54:44 smithi188 bash[77302]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-i 2024-04-13T03:54:45.301 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.i.service' 2024-04-13T03:54:45.335 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:45.336 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-13T03:54:45.336 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-13T03:54:45.336 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.l 2024-04-13T03:54:45.808 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:54:45 smithi188 systemd[1]: Stopping Ceph mds.l for e6bf0314-f944-11ee-bc8d-c7b262605968... 2024-04-13T03:54:45.809 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:54:45 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l[66805]: 2024-04-13T03:54:45.642+0000 7fe15b8b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-13T03:54:45.809 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:54:45 smithi188 ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l[66805]: 2024-04-13T03:54:45.642+0000 7fe15b8b6700 -1 mds.l *** got signal Terminated *** 2024-04-13T03:54:48.622 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:54:48 smithi188 podman[77424]: 2024-04-13 03:54:48.357069048 +0000 UTC m=+2.808879093 container died ad4dc24fcb9779a194092135dec14747953610fba512d528aa75524c00a9590b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-13T03:54:49.059 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:54:48 smithi188 podman[77424]: 2024-04-13 03:54:48.800708276 +0000 UTC m=+3.252518313 container cleanup ad4dc24fcb9779a194092135dec14747953610fba512d528aa75524c00a9590b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-13T03:54:49.059 INFO:journalctl@ceph.mds.l.smithi188.stdout:Apr 13 03:54:48 smithi188 bash[77424]: ceph-e6bf0314-f944-11ee-bc8d-c7b262605968-mds-l 2024-04-13T03:54:49.221 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e6bf0314-f944-11ee-bc8d-c7b262605968@mds.l.service' 2024-04-13T03:54:49.255 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-13T03:54:49.255 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-13T03:54:49.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 --force --keep-logs 2024-04-13T03:54:49.600 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:54:54.602 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 --force --keep-logs 2024-04-13T03:54:54.793 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:54:57.558 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 --force --keep-logs 2024-04-13T03:54:57.752 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T03:55:00.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-13T03:55:00.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-13T03:55:00.396 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-13T03:55:00.426 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-13T03:55:00.428 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/remote/smithi016/crash 2024-04-13T03:55:00.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/crash -- . 2024-04-13T03:55:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/crash: Cannot open: No such file or directory 2024-04-13T03:55:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-04-13T03:55:00.547 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/remote/smithi119/crash 2024-04-13T03:55:00.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/crash -- . 2024-04-13T03:55:00.608 INFO:teuthology.orchestra.run.smithi119.stderr:tar: /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/crash: Cannot open: No such file or directory 2024-04-13T03:55:00.608 INFO:teuthology.orchestra.run.smithi119.stderr:tar: Error is not recoverable: exiting now 2024-04-13T03:55:00.610 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/remote/smithi188/crash 2024-04-13T03:55:00.611 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/crash -- . 2024-04-13T03:55:00.650 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/crash: Cannot open: No such file or directory 2024-04-13T03:55:00.651 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2024-04-13T03:55:00.652 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-13T03:55:00.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-13T03:55:00.761 INFO:tasks.cephadm:Compressing logs... 2024-04-13T03:55:00.761 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-13T03:55:00.766 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-13T03:55:00.768 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-13T03:55:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-13T03:55:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:: No such file or directory 2024-04-13T03:55:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.l.log 2024-04-13T03:55:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.audit.log 2024-04-13T03:55:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.i.log 2024-04-13T03:55:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.l.log: /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.11.log 2024-04-13T03:55:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mgr.z.log 2024-04-13T03:55:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.i.log: /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.8.log 2024-04-13T03:55:00.797 INFO:teuthology.orchestra.run.smithi188.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-13T03:55:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.5.log 2024-04-13T03:55:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.f.log 2024-04-13T03:55:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.cephadm.log 2024-04-13T03:55:00.800 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.f.log: 90.4% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.audit.log.gz 2024-04-13T03:55:00.800 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log 2024-04-13T03:55:00.800 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.cephadm.log.gz 2024-04-13T03:55:00.800 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.2.log 2024-04-13T03:55:00.801 INFO:teuthology.orchestra.run.smithi119.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-13T03:55:00.804 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log: 87.2% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log.gz 2024-04-13T03:55:00.804 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.c.log 2024-04-13T03:55:00.805 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-13T03:55:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.10.log 2024-04-13T03:55:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.audit.log 2024-04-13T03:55:00.807 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.b.log 2024-04-13T03:55:00.807 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.4.log 2024-04-13T03:55:00.809 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.2.log: 92.2% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mgr.z.log.gz 2024-04-13T03:55:00.809 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-volume.log 2024-04-13T03:55:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.1.log 2024-04-13T03:55:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.k.log 2024-04-13T03:55:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.h.log 2024-04-13T03:55:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.7.log 2024-04-13T03:55:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.e.log 2024-04-13T03:55:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mgr.y.log 2024-04-13T03:55:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.cephadm.log 2024-04-13T03:55:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log 2024-04-13T03:55:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-volume.log 2024-04-13T03:55:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mon.b.log 2024-04-13T03:55:00.816 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-volume.log: /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mon.b.log: 92.2% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mgr.y.log.gz 2024-04-13T03:55:00.822 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mon.c.log 2024-04-13T03:55:00.838 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-13T03:55:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:: No such file or directory 2024-04-13T03:55:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.6.log 2024-04-13T03:55:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mon.a.log 2024-04-13T03:55:00.842 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.audit.log 2024-04-13T03:55:00.843 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.g.log 2024-04-13T03:55:00.845 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.j.log 2024-04-13T03:55:00.846 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.3.log 2024-04-13T03:55:00.847 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-volume.log: /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-volume.log.gz 2024-04-13T03:55:00.848 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mgr.x.log 2024-04-13T03:55:00.849 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.3.log: 90.2% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.audit.log.gz 2024-04-13T03:55:00.850 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.0.log 2024-04-13T03:55:00.852 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mgr.x.log: /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.0.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-13T03:55:00.854 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.cephadm.log 2024-04-13T03:55:00.856 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log 2024-04-13T03:55:00.857 INFO:teuthology.orchestra.run.smithi016.stderr: 94.4% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.g.log.gz 2024-04-13T03:55:00.857 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-volume.log 2024-04-13T03:55:00.857 INFO:teuthology.orchestra.run.smithi016.stderr: 84.0% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.cephadm.log.gz 2024-04-13T03:55:00.858 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.9.log 2024-04-13T03:55:00.861 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-volume.log: /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.d.log 2024-04-13T03:55:00.865 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.a.log 2024-04-13T03:55:00.871 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.a.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-13T03:55:00.871 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log.gz 2024-04-13T03:55:00.882 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-13T03:55:00.882 INFO:teuthology.orchestra.run.smithi016.stderr: 94.2% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.a.log.gz 2024-04-13T03:55:00.886 INFO:teuthology.orchestra.run.smithi016.stderr: 90.8% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-volume.log.gz 2024-04-13T03:55:00.948 INFO:teuthology.orchestra.run.smithi119.stderr: 90.9% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-volume.log.gz 2024-04-13T03:55:01.008 INFO:teuthology.orchestra.run.smithi119.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-13T03:55:01.025 INFO:teuthology.orchestra.run.smithi119.stderr: 90.4% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.audit.log.gz 2024-04-13T03:55:01.048 INFO:teuthology.orchestra.run.smithi119.stderr: 83.7% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.cephadm.log.gz 2024-04-13T03:55:01.050 INFO:teuthology.orchestra.run.smithi119.stderr: 87.1% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph.log.gz 2024-04-13T03:55:01.384 INFO:teuthology.orchestra.run.smithi016.stderr: 90.2% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mgr.x.log.gz 2024-04-13T03:55:02.362 INFO:teuthology.orchestra.run.smithi016.stderr: 90.3% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mon.a.log.gz 2024-04-13T03:55:02.422 INFO:teuthology.orchestra.run.smithi188.stderr: 92.7% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mon.c.log.gz 2024-04-13T03:55:03.642 INFO:teuthology.orchestra.run.smithi119.stderr: 91.7% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mon.b.log.gz 2024-04-13T03:55:03.674 INFO:teuthology.orchestra.run.smithi188.stderr: 95.1% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.l.log.gz 2024-04-13T03:55:03.877 INFO:teuthology.orchestra.run.smithi016.stderr: 95.1% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.j.log.gz 2024-04-13T03:55:04.394 INFO:teuthology.orchestra.run.smithi016.stderr: 95.2% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.d.log.gz 2024-04-13T03:55:06.655 INFO:teuthology.orchestra.run.smithi188.stderr: 95.1% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.f.log.gz 2024-04-13T03:55:15.888 INFO:teuthology.orchestra.run.smithi119.stderr: 95.1% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.k.log.gz 2024-04-13T03:55:24.605 INFO:teuthology.orchestra.run.smithi188.stderr: 91.4% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.11.log.gz 2024-04-13T03:55:25.239 INFO:teuthology.orchestra.run.smithi016.stderr: 91.4% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.0.log.gz 2024-04-13T03:55:28.684 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.6.log.gz 2024-04-13T03:55:29.294 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.3.log.gz 2024-04-13T03:55:30.739 INFO:teuthology.orchestra.run.smithi188.stderr: 90.9% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.8.log.gz 2024-04-13T03:55:31.685 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.9.log.gz 2024-04-13T03:55:31.690 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-13T03:55:31.690 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m30.888s 2024-04-13T03:55:31.690 INFO:teuthology.orchestra.run.smithi016.stderr:user 1m52.286s 2024-04-13T03:55:31.690 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m7.871s 2024-04-13T03:55:39.059 INFO:teuthology.orchestra.run.smithi188.stderr: 91.1% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.2.log.gz 2024-04-13T03:55:46.820 INFO:teuthology.orchestra.run.smithi188.stderr: 90.9% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.5.log.gz 2024-04-13T03:55:54.967 INFO:teuthology.orchestra.run.smithi188.stderr: 93.5% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.c.log.gz 2024-04-13T03:55:57.876 INFO:teuthology.orchestra.run.smithi188.stderr: 93.3% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.i.log.gz 2024-04-13T03:55:57.877 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-13T03:55:57.878 INFO:teuthology.orchestra.run.smithi188.stderr:real 0m57.096s 2024-04-13T03:55:57.878 INFO:teuthology.orchestra.run.smithi188.stderr:user 4m9.869s 2024-04-13T03:55:57.878 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m7.455s 2024-04-13T03:56:00.069 INFO:teuthology.orchestra.run.smithi119.stderr: 91.3% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.10.log.gz 2024-04-13T03:56:08.019 INFO:teuthology.orchestra.run.smithi119.stderr: 91.5% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.1.log.gz 2024-04-13T03:56:36.350 INFO:teuthology.orchestra.run.smithi119.stderr: 91.1% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.7.log.gz 2024-04-13T03:56:38.880 INFO:teuthology.orchestra.run.smithi119.stderr: 91.2% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-osd.4.log.gz 2024-04-13T03:57:33.499 INFO:teuthology.orchestra.run.smithi119.stderr: 93.4% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.e.log.gz 2024-04-13T03:57:46.751 INFO:teuthology.orchestra.run.smithi119.stderr: 93.4% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.h.log.gz 2024-04-13T03:58:28.587 INFO:teuthology.orchestra.run.smithi119.stderr: 93.3% -- replaced with /var/log/ceph/e6bf0314-f944-11ee-bc8d-c7b262605968/ceph-mds.b.log.gz 2024-04-13T03:58:28.677 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-13T03:58:28.677 INFO:teuthology.orchestra.run.smithi119.stderr:real 3m27.806s 2024-04-13T03:58:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:user 4m40.258s 2024-04-13T03:58:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m10.390s 2024-04-13T03:58:28.679 INFO:tasks.cephadm:Archiving logs... 2024-04-13T03:58:28.680 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/remote/smithi016/log 2024-04-13T03:58:28.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-13T03:58:46.411 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/remote/smithi119/log 2024-04-13T03:58:46.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-13T03:59:32.683 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/remote/smithi188/log 2024-04-13T03:59:32.684 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-13T04:00:07.368 INFO:tasks.cephadm:Removing cluster... 2024-04-13T04:00:07.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 --force 2024-04-13T04:00:07.714 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T04:00:08.686 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 --force 2024-04-13T04:00:08.977 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T04:00:09.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e6bf0314-f944-11ee-bc8d-c7b262605968 --force 2024-04-13T04:00:09.818 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: e6bf0314-f944-11ee-bc8d-c7b262605968 2024-04-13T04:00:10.375 INFO:tasks.cephadm:Removing cephadm ... 2024-04-13T04:00:10.376 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-13T04:00:10.439 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-13T04:00:10.455 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-13T04:00:10.470 INFO:tasks.cephadm:Teardown complete 2024-04-13T04:00:10.471 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-13T04:00:10.483 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-13T04:00:10.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-13T04:00:10.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-13T04:00:10.497 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-13T04:00:10.661 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-13T04:00:10.672 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-13T04:00:10.672 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T04:00:10.680 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T04:00:10.683 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-13T04:00:10.694 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-04-13T04:00:10.696 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-04-13T04:00:10.701 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T04:00:10.701 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-13T04:00:10.701 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +1105us[+1122us] +/- 49ms 2024-04-13T04:00:10.701 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -1099us[-1082us] +/- 71ms 2024-04-13T04:00:10.701 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 -513us[ -497us] +/- 67ms 2024-04-13T04:00:10.701 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T04:00:10.730 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-13T04:00:10.742 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T04:00:10.742 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-13T04:00:10.742 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv01.front.sepia.ceph.com 3 8 377 76 +1129us[+1160us] +/- 48ms 2024-04-13T04:00:10.742 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 80 -1077us[-1046us] +/- 70ms 2024-04-13T04:00:10.742 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 -523us[ -523us] +/- 67ms 2024-04-13T04:00:10.742 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T04:00:10.783 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-13T04:00:10.783 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-04-13T04:00:10.783 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv01.front.sepia.ceph.com 3 6 377 14 +1093us[+1086us] +/- 49ms 2024-04-13T04:00:10.783 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 13 -1113us[-1113us] +/- 71ms 2024-04-13T04:00:10.783 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -526us[ -526us] +/- 67ms 2024-04-13T04:00:10.783 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-13T04:00:10.794 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-13T04:00:10.805 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-13T04:00:10.805 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-13T04:00:10.877 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-13T04:00:11.134 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-13T04:00:11.232 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-13T04:00:11.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-13T04:00:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T04:00:11.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-13T04:00:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T04:00:11.447 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-13T04:00:11.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T04:00:11.477 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-13T04:00:11.488 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-13T04:00:11.498 INFO:teuthology.task.internal:Duration was 2859.338487 seconds 2024-04-13T04:00:11.499 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-13T04:00:11.509 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-13T04:00:11.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-13T04:00:11.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-13T04:00:11.520 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-13T04:00:11.563 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T04:00:11.583 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T04:00:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-13T04:00:12.202 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-13T04:00:12.202 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-04-13T04:00:12.203 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-13T04:00:12.270 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-04-13T04:00:12.270 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-13T04:00:12.311 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-04-13T04:00:12.311 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-13T04:00:12.336 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-13T04:00:12.336 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T04:00:12.343 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T04:00:12.354 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-13T04:00:12.443 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-13T04:00:12.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T04:00:12.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T04:00:12.451 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-13T04:00:12.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-13T04:00:13.002 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-13T04:00:13.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-13T04:00:13.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-13T04:00:13.162 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-13T04:00:13.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-13T04:00:13.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-13T04:00:13.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-13T04:00:13.213 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-13T04:00:13.254 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-04-13T04:00:13.255 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-04-13T04:00:13.325 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-04-13T04:00:13.385 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-13T04:00:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T04:00:13.437 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-13T04:00:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T04:00:13.451 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-13T04:00:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-13T04:00:13.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-13T04:00:13.477 INFO:teuthology.task.internal:Transferring archived files... 2024-04-13T04:00:13.478 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/remote/smithi016 2024-04-13T04:00:13.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-13T04:00:13.601 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/remote/smithi119 2024-04-13T04:00:13.602 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-13T04:00:13.639 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/remote/smithi188 2024-04-13T04:00:13.640 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-13T04:00:13.679 INFO:teuthology.task.internal:Removing archive directory... 2024-04-13T04:00:13.679 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-13T04:00:13.688 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-13T04:00:13.690 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-13T04:00:13.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-13T04:00:13.754 INFO:teuthology.task.internal:Not uploading archives. 2024-04-13T04:00:13.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-13T04:00:13.766 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-13T04:00:13.766 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-13T04:00:13.774 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-13T04:00:13.776 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-13T04:00:13.788 INFO:teuthology.orchestra.run.smithi119.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 04:00 /home/ubuntu/cephtest 2024-04-13T04:00:13.790 INFO:teuthology.orchestra.run.smithi188.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 04:00 /home/ubuntu/cephtest 2024-04-13T04:00:13.828 INFO:teuthology.orchestra.run.smithi016.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 13 04:00 /home/ubuntu/cephtest 2024-04-13T04:00:13.831 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-13T04:00:13.843 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-13T04:00:13.882 INFO:teuthology.nuke:Checking targets against current locks 2024-04-13T04:00:13.909 DEBUG:teuthology.nuke:shortname: smithi016 2024-04-13T04:00:13.909 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T04:00:13.933 DEBUG:teuthology.nuke:shortname: smithi119 2024-04-13T04:00:13.933 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T04:00:13.938 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 03:03:38.063441', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN2HJad3UK2cBZqWdWwY2LbNC8GcXG99vL24lyk30Is/SsXNDVuMnYZHJSRl8whGXG6DoJRrO9NSa4KD3lS1woQ='} 2024-04-13T04:00:13.957 DEBUG:teuthology.nuke:shortname: smithi188 2024-04-13T04:00:13.958 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-13T04:00:13.963 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 03:03:38.062041', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5f3IpZEfv1nVd7oXygMcKV7gLar+slPYHKPjyrp613EACIIN9rajZtVcHCdyTAgk2/cFKn+gjWIVsK8A3KPdY='} 2024-04-13T04:00:13.989 INFO:teuthology.orchestra.console.smithi016:Power off 2024-04-13T04:00:13.989 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-13T04:00:13.999 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-13 03:03:38.064737', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEP5S2oScurDuzg9YvE0nHj9+UqNo3pg9v+QZo/5JKTcZozh1+YnVjpf7V2j1oQicQ8rJESmTWTrLEMuVNrjBJc='} 2024-04-13T04:00:14.017 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2024-04-13T04:00:14.017 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:14.033 INFO:teuthology.orchestra.console.smithi119:Power off 2024-04-13T04:00:14.034 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-13T04:00:14.046 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-13T04:00:14.057 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-04-13T04:00:14.057 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:14.070 INFO:teuthology.orchestra.console.smithi188:Power off 2024-04-13T04:00:14.071 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-13T04:00:14.081 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-04-13T04:00:14.094 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-04-13T04:00:14.094 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:14.115 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-13T04:00:18.047 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:18.082 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:18.117 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:18.157 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-13T04:00:18.191 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-04-13T04:00:18.225 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-13T04:00:22.158 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:22.193 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:22.226 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:22.269 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2024-04-13T04:00:22.304 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-04-13T04:00:22.336 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-13T04:00:22.370 INFO:teuthology.orchestra.console.smithi016:Power off completed 2024-04-13T04:00:22.405 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-04-13T04:00:26.341 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-13T04:00:26.451 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-04-13T04:00:26.552 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-04-13T04:00:26.658 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} duration: 2859.3384866714478 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=df04e36084ae4a3cb22dd023563280d6 status: fail success: false 2024-04-13T04:00:26.659 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-13T04:00:26.760 INFO:teuthology.run:FAIL